In short
Builds and maintains scalable data pipelines supporting analytics and BI, processing large datasets with Python, PySpark, and SQL on Azure (Databricks, Data Factory). Collaborates with data scientists and analysts while monitoring pipeline performance and data quality.
About the role
We are seeking a skilled Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in Python, PySpark, and SQL, as well as experience with Azure technologies such as Databricks and Data Factory. As a Data Engineer, you will be responsible for designing, building, and maintaining scalable data pipelines that support our data analytics and business intelligence initiatives.
Key Responsibilities:
- Develop and maintain robust data pipelines using Python and PySpark to process large datasets efficiently.
- Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data solutions.
- Implement data integration solutions using Azure services, including Azure Data Factory and Databricks.
- Optimize data storage and retrieval processes to ensure high performance and reliability.
- Monitor and troubleshoot data pipeline performance, ensuring data quality and integrity.
- Stay updated with the latest industry trends and technologies in data engineering and cloud computing.
Required Skills:
- Proficiency in Python and PySpark for data processing and transformation.
- Strong SQL skills for querying and managing databases.
- Experience with Azure cloud services, particularly Databricks and Data Factory.
- Familiarity with big data technologies such as Spark.
- Excellent problem-solving skills and attention to detail.
- Ability to work collaboratively in a team environment and communicate effectively with stakeholders.
If you are passionate about data engineering and eager to work in a fast-paced environment, we encourage you to apply for this exciting opportunity.
Requirements
- Proficiency in Python and PySpark for data processing and transformation
- Strong SQL skills for querying and managing databases
- Experience with Azure cloud services, particularly Databricks and Data Factory