In short
Contract Data Engineer on a UK government data programme, building and running ELT pipelines into an S3-based data lake using Python, SQL, dbt, Airflow, Athena and Apache Iceberg, with Terraform and GitHub Actions CI/CD. Outside IR35, 6-month initial contract, hybrid working; SC clearance eligibility required.
About the role
Data Engineer - AWS Data Platform, Public Sector - SC eligible
About the role
Blackstone& is looking for a hands on Data Engineer to join a UK government data programme. The client runs an AWS data lake that analysts and data scientists rely on every day, and is moving more operational systems onto it. You'll help get data in reliably, model it well and keep it trustworthy.
You'll spend most of your time building, testing and running pipelines, working closely with analytics engineers, data scientists and the teams that own the source systems.
What you'll be doing
- Ingesting data from relational databases, file transfers and APIs into an S3 based data lake
- Building ELT pipelines with dbt and Amazon Athena, using Apache Iceberg tables
- Writing and scheduling workflows in Apache Airflow
- Using Python for pre processing, validation and automation, with tests in pytest
- Managing infrastructure as code with Terraform, and shipping through GitHub Actions CI/CD
- Building data quality checks so issues are caught before users see them
- Helping analysts and data scientists find, understand and use the data you deliver
- Keeping documentation current, so others can pick up your work
What we're looking for
- Strong Python and SQL
- Hands on AWS: S3, Athena, Glue, IAM & DMS
- dbt and Airflow, or close equivalents you can map across quickly
- Software engineering habits: git, code review, automated testing and CI/CD
- Able to explain trade offs to non technical colleagues
- Eligible for SC clearance
Nice to have:
- Terraform, Docker or Kubernetes
- Lambda or API Gateway for ingestion
- Spark or Glue PySpark for heavier workloads
- Experience in government or another highly regulated sector
Why this role
The stack is modern and open source, and the programme has backing to grow. The data you deliver will be used by analysts across the organisation to inform major public decisions.
ENGAGEMENT DETAILS
This engagement is offered outside IR35, reflecting the genuine independence and expertise the role requires. Key terms:
- Rate: Competitive
- Outside IR35 - engaged as an independent contractor via your own limited company or umbrella
- Initial contract length: 6 months, with extensions expected subject to performance and project progression
- Hybrid working - a mix of remote and on-site with the client as the engagement requires
- Start date: to be agreed. We are looking to move quickly for the right person