In short
Lead Data Engineer in Coimbatore designing and building scalable data platforms on Databricks, Apache Spark/PySpark, Python, SQL, Kafka, and Apache Pinot, covering real-time and batch pipelines, Lakehouse/Delta Lake architecture, plus solution architecture, pre-sales discussions, and mentoring the data engineering team.
About the role
Job Description Lead Data Engineer (Databricks)
Location: Coimbatore
Experience: 10–14 Years
Notice Period: Immediate Joiners Preferred
Role Overview
We are looking for an experienced Lead Data Engineer with strong expertise in Databricks, Apache Spark, PySpark, Python, SQL, Kafka, and Apache Pinot. The ideal candidate should have experience in solution architecture, data engineering, pre-sales discussions, and delivering scalable data solutions.
Key Responsibilities
- Lead the design and development of scalable data engineering solutions using Databricks.
- Design and implement data pipelines using Apache Spark, PySpark, Python, and SQL.
- Work on real-time and batch data processing using Kafka.
- Design and optimize low-latency data solutions using Apache Pinot.
- Implement Lakehouse architecture, Delta Lake, and Medallion architecture.
- Lead solution architecture discussions, POCs, and technical presentations.
- Collaborate with clients and stakeholders to understand business requirements.
- Drive technical decisions and mentor the data engineering team.
Mandatory Technical Skills
- Databricks
- Apache Spark / PySpark
- Python
- SQL
- Apache Kafka
- Apache Pinot
- Lakehouse Architecture
- Delta Lake / Medallion Architecture
Additional Skills
- Cloud platforms (AWS / Azure / GCP)
- Data pipeline orchestration (Airflow)
- DevOps and CI/CD
- Solution Architecture and Pre-sales
- Client-facing communication and stakeholder management
Required Experience
- 10–14 years of overall experience in Data Engineering.
- Strong hands-on experience with Databricks and Spark/PySpark.
- Experience in solution architecture and pre-sales activities.
- Experience in building scalable, high-performance data platforms.
- Strong communication and client-facing skills.
Preferred Location: Coimbatore
Note: Databricks and Apache Pinot are mandatory skills.