In short
Data engineer building an enterprise-scale AWS data platform in Abu Dhabi: day to day you design and run ETL/ELT and ingestion pipelines using Python/PySpark, SQL, AWS Glue, S3, Redshift, Athena, Lake Formation and Iceberg, with CI/CD and data-quality work. Minimum 5 years' experience; banking/financial-services background preferred.
About the role
- Strong hands-on experience delivering enterprise-scale data engineering solutions on AWS.
- Strong development experience with:
- Python / PySpark
- SQL
- AWS Glue
- AWS S3
- Strong experience designing and developing ETL/ELT and data ingestion pipelines.
- Experience working with:
- AWS Glue and Glue Catalog
- Amazon Redshift
- Amazon Athena
- AWS Lake Formation
- Apache Iceberg or similar open table formats
- Good understanding of:
- Data lake / lakehouse architecture
- Data warehousing
- Dimensional and analytical data modelling
- Data partitioning and optimisation
- Metadata management and data lineage
- Strong understanding of data quality, reconciliation and validation techniques.
- Experience integrating data from multiple source technologies, such as:
- Relational databases
- APIs
- Streaming/event-based sources
- Experience troubleshooting complex data and pipeline issues.
- Good understanding of IAM, security and data-access controls within AWS.
- Experience with version control and CI/CD using tools such as:
- GitHub
- Jenkins
- GitHub Actions
- Experience working within Agile delivery environments using JIRA and Confluence.
Minimum 5 yrs
Required Skills *- Strong hands-on experience delivering enterprise-scale data engineering solutions on AWS.
- Strong development experience with:
- Python / PySpark
- SQL
- AWS Glue
- AWS S3
- Strong experience designing and developing ETL/ELT and data ingestion pipelines.
- Experience working with:
- AWS Glue and Glue Catalog
- Amazon Redshift
- Amazon Athena
- AWS Lake Formation
- Apache Iceberg or similar open table formats
- Good understanding of:
- Data lake / lakehouse architecture
- Data warehousing
- Dimensional and analytical data modelling
- Data partitioning and optimisation
- Metadata management and data lineage
- Strong understanding of data quality, reconciliation and validation techniques.
- Experience integrating data from multiple source technologies, such as:
- Relational databases
- APIs
- Files / object storage
- Streaming/event-based sources
- Experience troubleshooting complex data and pipeline issues.
- Good understanding of IAM, security and data-access controls within AWS.
- Experience with version control and CI/CD using tools such as:
- GitHub
- Jenkins
- GitHub Actions
- AWS CodePipeline
- Experience working within Agile delivery environments using JIRA and Confluence.
- Strong professional experience in Data Engineering and Enterprise Data Platforms.
- Demonstrable experience delivering production-grade data pipelines in complex enterprise environments.
- Experience working within Banking or Financial Services strongly preferred.
- Understanding of data governance, security and regulatory requirements within a controlled enterprise environment.
- Experience working across the full engineering lifecycle from requirements and source-data analysis through development, testing, deployment and production support.