About the role
Casado is hiring a senior Data Engineer to design, build, and optimize data pipelines on Azure Databricks and PySpark. The role focuses on reliable ingestion, transformation, and governance across ADLS Gen2 and Delta Lake in a cloud data lakehouse.
You will implement Medallion architecture, enhance CI/CD practices with Azure DevOps, and collaborate with architects, analysts, and BI teams in an Agile setting to meet SLAs.
Design, develop, and maintain data pipelines using Azure Databricks and PySpark. Build and orchestrate ETL/ELT workflows with ADF. Develop ingestion & transformation pipelines using ADLS Gen2, Delta Lake, and Spark SQL. Implement Medallion architecture (Bronze/Silver/Gold), incremental loads, CDC, MERGE, and SCD Type 1/2. Optimize Spark jobs via partitioning, caching, file-size optimization, and performance tuning. Configure and maintain Databricks Workflows/Jobs, clusters, and workspaces. Build data quality checks, validation, error handling, monitoring, and reconciliation. Leverage Unity Catalog for governance, lineage, RBAC, and secure data access. Integrate Databricks with Azure services (ADLS, Azure SQL, APIs, JDBC). Implement CI/CD with Azure DevOps & Git; automate IaC with Terraform or ARM/Bicep. Troubleshoot production pipelines, run root-cause analysis, and ensure SLA compliance. Collaborate with architects, analysts, and BI teams in an Agile environment. 7–10+ years in Data Engineering or Azure cloud data development. Hands-on with Azure Databricks, PySpark, Spark SQL, and Delta Lake. Proven experience building scalable ETL/ELT pipelines with ADF. Solid grasp of ADLS Gen2, Lakehouse architecture, and distributed data processing. CI/CD with Azure DevOps, Git, YAML pipelines, and IaC (Terraform/ARM/Bicep). Strong SQL skills with performance-tuning expertise. Experience in Agile/Scrum environments. Databricks platform access