About the role
Prosus is seeking a senior data engineer to build and operate the LCM training-data platform, handling acquisition, processing, synthetic generation, quality evaluation and dataset versioning. You will work with scientists and ML engineers to turn data into reproducible datasets and measure how data changes affect model performance.
You will implement pipelines and tooling for data provenance, tagging, evaluation and cost-aware operation, enabling researchers to run training runs with traceable
Build acquisition and processing pipelines for portfolio data and external sources. Develop the shared platform for inspecting sources, configuring processing and publishing training datasets. Operate distributed workflows, monitor reliability and cost, implement access controls and retention requirements. Work with scientists, engineers and teams on data provenance and privacy-aware handling of histories. Implement dataset versioning and lineage across sources, processing code and dataset configurations. Build quality checks for completeness, freshness, duplication, extraction accuracy and coverage. Provide tools for source tagging, sampling and weighting, and maintain evaluation datasets with temporal splits and overlap checks. At least five years of relevant data engineering experience. Strong Python and SQL, with experience writing tested, maintainable code. Hands-on experience with distributed processing using Spark, Ray, Beam.Skills
Interested in this role?
You'll apply directly on Prosus's careers site.