About the role
We are looking for a hands-on Data Engineer to help build and manage our data platform for reporting, analytics, and future data science use cases.
The role will involve working with SQL, Python, PySpark, AWS, ETL/ELT pipelines, data warehouses, and BI/reporting tools. Our architecture may use technologies such as ClickHouse, Redshift, AWS Glue, Airflow, Step Functions, Lambda, S3, and CDC-based replication tools based on scale, cost, and operational needs.
Key Responsibilities
· Design, build, and maintain scalable ETL/ELT data pipelines from multiple databases, applications, and external systems.
· Build raw, cleaned, and business-ready data layers to support reporting, analytics, and future data science use cases.
· Write efficient SQL, Python, and PySpark jobs for data ingestion, transformation, validation, and processing.
· Implement workflow orchestration using Apache Airflow, AWS Glue, Step Functions, or similar tools.
· Work with data warehouses such as ClickHouse, Redshift, Snowflake, BigQuery, or similar.
· Support cross-service reporting as the architecture moves towards independent microservice databases.
· Build reusable reporting tables, aggregates, summaries, and basic data marts.
· Monitor, troubleshoot, and optimize data pipelines, warehouse queries, and processing jobs.
· Implement data quality checks for freshness, completeness, consistency, duplicates, and reconciliation.
· Support BI/reporting needs through tools such as Power BI, Metabase, Superset, Redash, or similar.
· Apply data governance, access control, security, and PII-handling best practices.
· Collaborate with engineering, DevOps, product, business, finance, risk, and support teams.
Required Skills
· Strong expertise in SQL for joins, aggregations, window functions, query optimization, and analytical reporting.
· Hands-on experience with Python for data processing, automation, validation, and scripting.
· Working experience with PySpark / Apache Spark for processing large datasets.
· Good understanding of ETL/ELT pipelines, data warehousing, and data modelling concepts.
· Experience with workflow orchestration using Apache Airflow, AWS Step Functions, AWS Glue, or similar tools.
· Experience with AWS data services such as S3, Glue, Lambda, Step Functions, Redshift, DMS, CloudWatch, or similar.
· Experience with any data warehouse such as ClickHouse, Redshift, Snowflake, BigQuery, or similar.
· Understanding of relational databases, preferably PostgreSQL.
· Ability to debug data mismatches, failed pipelines, slow queries, and data quality issues.
· Exposure to BI tools such as Power BI, Metabase, Superset, Redash, or similar.
Good ownership, problem-solving, communication, and collaboration skills.
Skills:- Data engineering, ClickHouse, Amazon Web Services (AWS) and PySpark