In short
Data Engineer building and maintaining scalable data pipelines and architectures supporting analytics/BI, using Apache Spark/PySpark, Apache Airflow, SQL, and AWS cloud services. Collaborates with data scientists and analysts to deliver high-quality data solutions.
About the role
We are seeking a skilled Data Engineer to join our dynamic team. The ideal candidate will have a strong background in designing and implementing data pipelines, working with big data technologies, and utilizing cloud services. As a Data Engineer, you will be responsible for developing and maintaining robust data architectures that support our analytics and business intelligence initiatives.
Key Responsibilities:
- Design, build, and maintain scalable data pipelines using Apache Spark and PySpark.
- Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data solutions.
- Implement data ingestion processes using Apache Airflow to automate workflows and ensure timely data availability.
- Optimize SQL queries for performance and efficiency in data retrieval and processing.
- Work with AWS cloud services to deploy and manage data solutions in a cloud environment.
- Ensure data quality and integrity through rigorous testing and validation processes.
- Stay updated with the latest industry trends and technologies in big data and cloud computing.
Required Skills:
- Proficiency in Apache Spark and PySpark for data processing.
- Strong programming skills in Python.
- Experience with SQL for database management and data manipulation.
- Familiarity with data pipeline tools and frameworks, particularly Apache Airflow.
- Knowledge of big data technologies and concepts.
- Experience with AWS cloud services and data bricks is a plus.
If you are passionate about data engineering and eager to work in a fast-paced environment, we encourage you to apply. Join us in transforming data into actionable insights that drive business success.
Requirements
- Proficiency in Apache Spark and PySpark for data processing
- Strong programming skills in Python
- Experience with SQL for database management and data manipulation