In short
Remote Data Engineer at LCG Inc. (Portugal) designing, developing and maintaining data pipelines with Azure Databricks, PySpark and Azure Data Factory. Day to day involves ingesting, transforming and modeling data in a Data Lakehouse, preparing ML datasets, training/evaluating models and monitoring data quality.
About the role
A LCG Inc. procura um Data Engineer para integrar a equipa em regime remoto.
O papel centra-se em desenhar, desenvolver e manter pipelines de dados eficientes, usando Azure Databricks, PySpark e Azure Data Factory conforme necessário. As responsabilidades incluem ingesto, transformação e modelação de dados num Data Lakehouse, preparação de datasets para Machine Learning, treino e avaliação de modelos, e monitorização de qualidade de dados.
Requirements
- Design, develop and maintain efficient data pipelines
- Azure Databricks
- PySpark
- Data ingestion, transformation and modeling in a Data Lakehouse
- Preparing datasets for Machine Learning
- Training and evaluating ML models
- Data quality monitoring
Nice to have
- Azure Data Factory