About the role
Job Description – Databricks Data Engineer
Job Title: Databricks Data Engineer
Experience: 5–8 Years
Job Summary:
We are looking for an experienced Databricks Data Engineer with strong expertise in PySpark, Python, and SQL to develop data profiling notebooks and build scalable data processing solutions. The ideal candidate should have hands-on experience with Azure Databricks, data profiling, data quality assessment, and data validation, along with working knowledge of Power BI and Fivetran.
Key Responsibilities:
- Develop and maintain data profiling notebooks using Databricks, PySpark, Python, and SQL.
- Perform data profiling, data quality assessment, validation, and cleansing across multiple data sources.
- Build and optimize data transformation and processing workflows using PySpark and SQL.
- Analyze datasets to identify data inconsistencies, duplicates, missing values, and anomalies.
- Develop reusable notebooks for data profiling, quality checks, and validation.
- Work on data ingestion and integration using Fivetran.
- Collaborate with data engineering and analytics teams to support data pipeline development.
- Create and maintain data quality reports and dashboards using Power BI.
- Optimize Databricks notebooks and queries for performance and scalability.
- Document data profiling results, findings, and recommendations.
Required Technical Skills
- 5–8 years of experience in Data Engineering.
- Strong hands-on experience with Azure Databricks.
- Strong proficiency in PySpark, Python, and SQL.
- Experience developing Databricks notebooks for data profiling and validation.
- Good understanding of data quality, data cleansing, and data transformation.
- Experience working with large datasets and multiple data sources.
- Working knowledge of Power BI for reporting and visualization.
- Working knowledge of Fivetran for data ingestion and integration.
- Understanding of ETL/ELT processes and data engineering concepts.