About the role
We're Hiring: Spark Snowflake Engineer
Location- Remote
Are you passionate about building scalable data solutions and driving cloud data transformation initiatives? We are looking for a highly skilled Spark Snowflake Engineer with 8+ years of experience in Data Engineering, Data Warehousing, Big Data Processing, Cloud Data Platforms, and Enterprise Data Modernization.
Key Responsibilities:
- Design, develop, and optimize large-scale data pipelines and ETL/ELT processes.
- Lead and support Enterprise Data Warehouse (EDW) migration and modernization initiatives.
- Build high-performance data solutions using Apache Spark and Snowflake.
- Develop and integrate solutions using Python and REST APIs.
- Collaborate with cross-functional teams to deliver cloud-based data transformation programs.
- Ensure data quality, scalability, security, and performance across data platforms.
Key Responsibilities
Spark Snowflake Data Engineer (Python)
- Design, develop, and maintain scalable ETL/ELT pipelines using Python, Spark, and Snowflake.
- Build ingestion frameworks to load data from multiple source systems into cloud-based data platforms.
- Translate business and technical requirements into production-ready solutions.
- Develop reusable data processing components, frameworks, and automation utilities.
- Implement data transformation, cleansing, enrichment, and validation processes.
- Ensure data pipelines are scalable, reliable, maintainable, and high performing.
- Troubleshoot and resolve data integration and processing issues.
Python, Spark & Scala Development
- Develop distributed data processing applications using Python, PySpark, and Scala.
- Build scalable Spark-based transformation and processing pipelines.
- Optimize Spark workloads for resource utilization and performance.
- Implement logging, monitoring, error handling, and exception management frameworks.
- Develop reusable code libraries and common framework components.
- Participate in performance tuning and application optimization initiatives.
API Development & Integration
- Design, develop, and maintain RESTful APIs and integration services.
- Enable API-based data exchange across enterprise systems.
- Implement API authentication, authorization, and security standards.
- Create and maintain API documentation.
- Support API lifecycle and version management.
- Collaborate with integration teams to ensure seamless enterprise-wide data consumption.
Data Migration & Modernization
- Participate in Enterprise Data Warehouse (EDW) migration and modernization initiatives.
- Support migration from legacy platforms including:
Teradata, Netezza, Oracle, SQL Server, Hadoop
- Develop source-to-target mappings and transformation logic.
- Build ingestion and migration pipelines into Snowflake.
- Perform data validation, reconciliation, and migration testing.
- Support deployment, cutover, and post-migration stabilization activities.
Snowflake Development
- Design, develop, and maintain Snowflake-based data solutions.
- Build scalable data ingestion and transformation frameworks.
- Develop and optimize:
Snowflake Tables, Views, Streams, Tasks, Snowpipe, Stored Procedures
- Support enterprise data warehouses and data marts.
- Monitor and optimize Snowflake query performance, storage, and compute utilization.
- Implement secure, governed, and role-based data access mechanisms.