About the role
Inetum in Madrid is seeking a senior Data Engineer to migrate Hadoop infrastructure to cloud-based Kubernetes, Spark on Scala, and Airflow as a service. You will design data transformations, ensure data quality, and collaborate with cross-functional teams to deliver scalable data solutions.
The role requires strong SQL/NoSQL skills, knowledge of HDFS, S3/COS, Parquet/ORC formats, and experience with CI/CD pipelines, testing, and documentation.
Migrate the Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service. Implement data transformation and quality to ensure data consistency and accuracy using Spark/Scala and SQL. Set up CI/CD pipelines to automate deployments, unit testing and development management. Write and conduct unit and validation tests to ensure accuracy and integrity of code developed. Automate data pipelines and streamline data ingestion with orchestrators (Airflow as a Service). Write technical documentation to ensure knowledge capitalization. Collaborate with cross-functional teams to understand data requirements and deliver solutions. Strong knowledge of Spark on Scala. Proficient in Hadoop, HDFS and SQL/NoSQL databases. Experience with cloud data services (COS/S3) and Parquet/ORC formats. Shell scripting and SDLC awareness; Agile experience. Ability to design data transformations and data quality processes.