In short
Hands-on Data Engineer at Accenture in Melbourne building real-time data pipelines that move high-volume data from Amazon Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables. Day to day involves integrating streaming and batch workflows with cloud data teams and writing clean Python and SQL while optimising for performance, latency and cost.
About the role
Accenture Global in Melbourne is seeking a hands-on Data Engineer to build reliable real-time data pipelines using Amazon Kinesis and Databricks. You will move high-volume data from Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.
You’ll collaborate with cloud data teams to ensure streaming and batch processes are well-integrated, write clean Python and SQL, and optimise pipelines for performance, latency and cost in a fast-paced environment.
Requirements
- Hands-on data engineering experience
- Building real-time data pipelines with Amazon Kinesis
- Databricks Lakehouse
- Spark Structured Streaming
- Delta tables
- Writing clean Python
- Writing clean SQL
- Integrating streaming and batch processes with cloud data teams
- Optimising pipelines for performance, latency and cost