About the role
Loveyourdevteam seeks a Data Engineer to design and operate high-performance data pipelines in Python, leveraging Medallion architecture to transform raw data into analytics-ready assets. The role focuses on standardized storage with Parquet, Iceberg, and Delta Lake and embedding fast analytics with DuckDB within ETL workflows.
Responsibilities include optimizing PostgreSQL schemas, using SQLite for lightweight storage, and scaling ETL with Databricks or PySpark in distributed environments.
Pipeline Design: Build resilient data pipelines in Python (Medallion architecture). Open Storage Management: Standardize storage with Parquet, Iceberg, Delta Lake. Embedded Analytics: Use DuckDB within ETL workflows for fast processing. Database Operations: Architect/optimize PostgreSQL schemas; use SQLite for embedded storage. Distributed Compute: Scale ETL workloads with Databricks or Apache Spark. Advanced SQL and PostgreSQL expertise required. Experience building analytical pipelines with Python and DuckDB. Proficiency with open storage formats (Parquet, Iceberg, Delta Lake). Experience designing data lakes using Medallion architecture. Familiarity with Databricks/PySpark for distributed processing.