About the role
ROLE SUMMARY :
We are looking for a hands-on Data Engineering Manager with strong expertise in Python, PySpark, SQL, and Apache Airflow. The role will lead a team of approximately 8 - 12 engineers, own delivery of scalable data platforms and pipelines, provide technical direction, and remain hands-on across architecture, development, performance optimization, data quality, and production support.
KEY RESPONSIBILITIES :
- Lead the architecture, design, and delivery of scalable batch and near-real-time data pipelines using Python and PySpark.
- Provide technical direction for Apache Airflow orchestration, including DAG standards, scheduling, dependencies, monitoring, retries, and recovery.
- Guide development of optimized SQL for ingestion, transformation, reconciliation, controls, and analytical workloads.
- Manage and mentor 8 - 12 data engineers while remaining hands-on with design reviews, code reviews, debugging, and complex problem solving.
- Own delivery planning, estimation, execution, quality, risks, dependencies, release readiness, and production stability.
- Drive data-quality controls, lineage, observability, security, auditability, and operational support standards.
- Optimize Spark jobs, SQL queries, partitioning, caching, joins, resource utilization, cloud performance, and cost.
- Partner with Product Owners, Architects, Governance, DevOps, QA, and client stakeholders to deliver measurable business outcomes.
REQUIRED SKILLS :
- 10+ years of data or software engineering experience, including proven technical, delivery, and people leadership.
- Strong hands-on Python with OOP, modular architecture, testing, exception handling, and reusable framework development.
- Deep PySpark expertise across DataFrames, Spark SQL, distributed processing, partitioning, joins, caching, and performance tuning.
- Advanced SQL including CTEs, window functions, complex joins, validation, reconciliation, data modelling, and query optimization.
- Strong Apache Airflow expertise covering DAG architecture, operators, sensors, dependencies, scheduling, monitoring, and troubleshooting.
- Experience with relational databases, data lakes or warehouses, Parquet, schema evolution, and high-volume data processing.
- Experience with Git, CI/CD, Linux, test automation, deployment governance, and production support practices.
- Strong Agile delivery, stakeholder management, communication, risk management, and team-development skills.
PREFERRED SKILLS :
- Financial Services / Capital Markets experience, including reconciliation, controls, risk, or regulatory data.
- Azure or AWS data services, Databricks, Snowflake, Docker, Kubernetes, or Kafka exposure.
- Knowledge of data governance, metadata management, security, observability, and cloud cost optimization.
IDEAL CANDIDATE PROFILE :
Hands-on Data Engineering Manager who can lead teams and delivery while architecting reliable pipelines, optimizing distributed workloads, strengthening data quality, and owning production outcomes.
ROLE POSITIONING :
Technical Leadership + Data Engineering Delivery + People Management