Senior Data Engineer

About the role

DataHub Analytics is seeking an experienced Senior Data Engineer to design, develop and optimise scalable data pipelines that support its regional expansion and AI‑driven initiatives. The role focuses on building reliable, high‑performance data solutions across structured and unstructured sources.

Key responsibilities

• Design, develop and maintain scalable ETL/ELT pipelines.
• Build and optimise data processing solutions using Python and Apache Spark.
• Develop real‑time and near‑real‑time pipelines with Apache Kafka.
• Write and optimise complex SQL queries and transformation logic.
• Integrate data from multiple structured and unstructured sources.
• Ensure data quality, reliability and performance in large‑scale environments.
• Troubleshoot pipeline failures, performance bottlenecks and data inconsistencies.
• Implement monitoring, logging, error handling and recovery mechanisms.
• Collaborate with data architects, analysts, developers and business stakeholders.
• Participate in data modelling, solution design, code reviews and technical discussions.

Required profile

• 4+ years of professional experience in data engineering, ETL or data integration.
• Strong hands‑on experience developing ETL/ELT pipelines.
• Proficiency in Python and SQL.
• Hands‑on experience with Apache Spark and distributed data processing.
• Experience with Apache Kafka or similar event‑streaming technologies.
• Solid understanding of relational databases, data warehouses and data modelling concepts.
• Experience troubleshooting and optimising production data pipelines.
• Strong analytical, problem‑solving and communication skills.

Required skills

• Python
• Apache Spark
• Apache Kafka
• SQL
• Java (nice to have)
• Apache NiFi (nice to have)
• Cloud platforms: AWS, Azure, GCP (nice to have)
• Apache Airflow (nice to have)