In short
Builds and maintains scalable data pipelines in Databricks/Apache Spark, developing ETL/ELT workflows with Python, PySpark, SQL, and Delta Lake on cloud platforms like Azure, AWS, or GCP. Focuses on data ingestion, performance tuning, and data quality.
About the role
Key Responsibilities :
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Build and optimize ETL/ELT workflows for processing structured and unstructured data.
- Develop data transformation and processing logic using Python, PySpark, and SQL.
- Work with Delta Lake to manage reliable, scalable, and high-performance data storage.
- Implement data ingestion solutions from multiple sources, including databases, APIs, files, and cloud storage.
- Optimize Spark jobs, queries, and data pipelines for performance, scalability, and cost efficiency.
- Implement data quality checks, validation rules, and monitoring mechanisms.
- Collaborate with Data Architects, Data Scientists, Business Analysts, and other engineering teams to understand data requirements.
- Work with cloud-based data platforms and services to build integrated data solutions.
- Follow best practices for data security, governance, version control, and deployment.
- Troubleshoot data pipeline failures and ensure data availability, accuracy, and reliability.
Required Skills & Qualifications :
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- Hands-on experience with Azure Databricks or Databricks.
- Strong programming skills in Python and PySpark.
- Strong proficiency in SQL and experience working with relational databases.
- Good understanding of Apache Spark architecture and distributed data processing.
- Experience developing and managing ETL/ELT data pipelines.
- Hands-on experience with Delta Lake and Delta tables.
- Understanding of data warehousing concepts, data modelling, and data integration.
- Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP.
- Familiarity with Git and CI/CD practices for data engineering workflows.
- Strong analytical, problem-solving, and debugging skills.
Requirements
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field