About the role
The Role
You will design, build, secure, automate, and operate enterprise Databricks Lakehouse platforms for banking and financial services clients, ensuring scalability, reliability, governance, and performance across multiple environments and regions.
Key Responsibilities
- Build and manage enterprise Databricks platforms across Dev, Test, UAT, and Production environments.
- Provision and configure Databricks workspaces, clusters, compute resources, and platform settings.
- Automate infrastructure provisioning and platform configuration using Terraform and IaC.
- Develop and maintain CI/CD pipelines for Databricks deployments, notebooks, jobs, and configurations.
- Administer Unity Catalog, including catalogs, schemas, permissions, and data access controls.
- Implement security controls, RBAC, service principals, secrets management, and least-privilege access.
- Integrate Databricks with AWS/Azure services, including storage, networking, identity, and security.
- Monitor platform health, troubleshoot production issues, and implement logging, alerting, and performance optimisation.
- Optimise compute resources and automate platform operations for cost efficiency and reliability.
- Collaborate with Data Engineering, Cloud, DevOps, Security, and Governance teams to deliver secure, scalable platforms.
Your Profile
Databricks & Lakehouse
- Strong hands-on experience with Databricks, including Notebooks, PySpark, SQL, Delta Lake, Jobs, Workflows, and Declarative Pipelines.
- Experience with Unity Catalog, Databricks Auto Loader, and Structured Streaming.
- Experience developing batch and incremental data pipelines.
- Strong understanding of Lakehouse architecture, including Bronze, Silver, and Gold layers.
AWS & Cloud
- Good knowledge of AWS services, particularly S3, IAM, KMS, Glue, and Lambda.
- Experience migrating data workloads from AWS-native services to Databricks.
- Understanding of cloud infrastructure, security, networking, and identity management.
DevOps & Data Engineering
- Experience with Git, CI/CD pipelines, and deployment automation.
- Knowledge of Terraform or Infrastructure as Code (IaC) is desirable.
- Strong understanding of distributed data processing, data modelling, and performance optimisation.
Requirements
- Strong hands-on experience with Databricks, including Notebooks, PySpark, SQL, Delta Lake, Jobs, Workflows, and Declarative Pipelines.