About the role
Summary:
We are seeking a seasoned Senior Data Engineer to drive the development and deployment of robust data solutions that foster organizational growth and unlock new market opportunities. This pivotal role involves leading a diverse team to architect and execute innovative analytical strategies that deliver tangible business value.
Responsibilities:
- Design and deploy scalable data pipelines utilizing various formats such as JSON, Parquet, CSV, and ORC for both batch and streaming ingestion.
- Spearhead cloud migration initiatives by developing high-performance Spark pipelines that ensure efficiency and reliability.
- Architect and maintain Medallion Architecture frameworks, including the implementation of Bronze, Silver, and Gold tables for structured data systems.
- Optimize Spark codebases to enhance performance during cloud transitions and overall system efficiency.
- Establish and enforce robust data modeling practices that adhere to strict governance standards.
- Implement comprehensive data cataloging strategies using Unity Catalog to guarantee data integrity and quality.
- Lead the creation and management of Delta Live Tables (DLT) pipelines to ensure secure and tamper-resistant data operations.
- Partner with clients to refine cloud migration strategies and enforce industry best practices in design and governance.
Requirements:
- Possess a minimum of five years of hands-on experience in data engineering with a demonstrated history of managing complex pipeline development and cloud migration projects.
- Hold a Bachelor's degree or higher in Computer Science, Data Engineering, or a closely related field.
- Demonstrate advanced proficiency in Spark, SQL, and Python, along with other essential data processing technologies.
- Show deep expertise in optimizing on-premises Spark code for cloud environments and implementing Medallion Architecture patterns.
- Maintain strong working knowledge of Databricks ecosystems, including specific experience with Delta Live Table (DLT) implementations.
- Possess solid familiarity with AWS services, with additional exposure to GCP or Azure platforms being highly advantageous.
- Exhibit excellent communication and collaboration abilities to effectively engage with both clients and internal cross-functional teams.
- Hold relevant professional certifications in Data Engineering for AWS, GCP, or Azure as a preferred qualification.