In short
A senior data engineer in Sydney designing, building and optimising high-throughput ETL/ELT pipelines with Azure Databricks, Spark (Scala), SQL and Python, plus real-time integrations and CI/CD, scheduling, secrets management and observability across Azure and AWS.
About the role
Showtime Consulting Pty Ltd in Sydney seeks an experienced Data Engineer to join a large enterprise data environment. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.
You will deliver scalable ETL/ELT pipelines, support real-time integrations, and operationalise workloads through CI/CD, scheduling, secrets management and observability across cloud platforms such as Azure and AWS.
Requirements
- Experience as a (senior) Data Engineer
- Design, build and optimise high-throughput data pipelines using Azure Databricks
- Spark with Scala
- SQL
- Python
- Deliver scalable ETL/ELT pipelines
- Support real-time integrations
- Operationalise workloads via CI/CD, scheduling, secrets management and observability
- Experience with cloud platforms such as Azure and AWS