In short
Remote data-engineering role rebuilding legacy T-SQL logic as typed, tested PySpark pipelines on Delta Lake, plus FastAPI services to run and report on pipeline jobs and provider-neutral LLM features. Core stack: Python 3.12, Spark/Spark SQL, Delta Lake, FastAPI, pytest, Docker, Azure.
About the role
Role Overview:
- Rebuild T-SQL logic as typed, tested PySpark pipelines.
- Prove that new pipelines match legacy outputs.
- Build FastAPI services that run and report on the pipeline jobs.
- Add provider-neutral LLM features into the data workflows.
Pipeline Responsibilities:
- Build Spark and PySpark pipelines on Delta Lake.
- Re-express T-SQL logic in Spark SQL.
- Ensure schema evolution, time travel, and idempotent writes using Delta Lake.
- Optimize partitioning and performance for production Spark jobs.
API and Service Development:
- Build REST APIs with FastAPI for pipeline jobs.
- Implement request validation, authentication, and job-status semantics.
- Ensure idempotency in service architecture.
Code Quality and Standards:
- Write type-hinted, tested Python code using Python 3.12, Pydantic, and static analysis tools like pyright or mypy.
- Implement automated testing with pytest, including fixtures, markers, and tiered test suites.
- Utilize modern packaging tools such as uv or Poetry.
- Use Git and CI discipline with PR-driven workflows, including ruff and linting gates.
- Work against design documents and Architecture Decision Records.
- Maintain documentation alongside every change.
Data Migration and Validation:
- Validate migrated pipelines against legacy systems with parity evidence.
- Read T-SQL and reproduce its semantics faithfully in set-based logic.
- Manage migration of legacy SQL Server or Azure Synapse estates to Spark.
Development Environment:
- Use Docker-based development against Compose stacks.
- Work autonomously from written requirements and specifications.
Preferred Skills and Experience:
- Integrate provider-neutral AI features and validate structured outputs.
- Implement data governance and privacy engineering such as PII tokenization and GDPR-style data handling.
- Maintain Lakehouse platform components like Hive Metastore, Trino, or Apache Ranger.
- Manage Azure data estate components including ADLS Gen2, Synapse, or Key Vault.
- Utilize observability tooling such as structured logging, OpenTelemetry, and OpenLineage.
Location: Remote - Portugal, Spain, Italy, Greece, France, Belgium, Netherlands, Germany, Poland, Norway, Sweden, Finland, Estonia, Lithuania, Latvia, Czechia, Slovakia, Austria, Hungary, Slovenia
Skills required for this job:
- Apache Spark
- Artificial intelligence (AI)
- Python