About the role
Location - Chennai
Role Summary:
We are seeking an experienced Lead Data Engineer (Level 4) to serve as the technical leader responsible for the modernization and future-state architecture of our enterprise Sales Validation platform. In this role, you will lead the end-to-end technical delivery, drive a critical platform migration from Azure Synapse to Databricks, establish engineering excellence standards, and mentor high-performing data engineering teams.
Key Responsibilities:
- Architecture & Modernization: Lead Databricks architecture, future-state design, and delivery across major platform workstreams.
- Platform Migration: Direct and execute the end-to-end migration of legacy ETL/ELT workloads from Azure Synapse to Databricks.
- Engineering Standards & Frameworks: Establish robust engineering standards, design reusable frameworks, and implement best practices for Lakehouse architecture.
- Technical Ownership & Optimization: Own technical delivery across key workstreams, driving performance tuning, scalability, and system reliability.
- Leadership & Quality Assurance: Conduct thorough architecture/design reviews, enforce code quality via code reviews, and mentor junior to mid-level engineers.
Required Qualifications & Technical Skills:
Databricks & Lakehouse (Core Focus):
- Deep expertise in Databricks and Lakehouse Architecture.
- Advanced hands-on experience with PySpark, Spark SQL, and Delta Lake.
- Production experience implementing Unity Catalog for governance and access management.
- Strong knowledge of orchestration and automation using Databricks Workflows and Databricks Asset Bundles (DABs).
Azure Data Platform:
- Proven experience building and orchestrating pipelines using Azure Data Factory (ADF).
- Strong knowledge of Azure Data Lake Storage Gen2 (ADLS Gen2), Azure SQL, and enterprise Azure Security Patterns.
Migration & Legacy Modernization:
- Hands-on experience modernizing legacy ETL workloads and migrating from Azure Synapse Analytics to Databricks.
- Deep practical knowledge of Synapse components: Synapse Pipelines, Synapse Spark, Synapse SQL Pools, Synapse Notebooks, and Synapse Data Flows.
Core Software & Data Engineering:
- Advanced SQL and complex Data Modeling capabilities.
- Proven track record in query and job performance tuning and optimization.
- Experience with CI/CD automation and version control using GitLab CI/CD.