In short
Big Data Engineer developing new features in Spark/Scala on an on-premises Cloudera Hadoop platform hosting a Finance and Risk data lake, in an Agile/DevOps context. Core stack: Spark 3, Scala 2.12, HDFS/Hive/HBase/Kafka secured with Kerberos/Ranger, plus Jenkins/XL Deploy CI/CD and Control-M scheduling.
About the role
Role overview
We are looking for a Big Data Engineer to strengthen projects built on an on-premises Cloudera Hadoop Big Data platform that hosts a Finance and Risk data lake. The role focuses on designing and developing new features in Spark/Scala within an Agile and DevOps context, with a strong emphasis on code quality, reuse, and scalable architecture.
Key responsibilities
- Take part in the technical design of the features to be developed.
- Develop new features in Spark/Scala.
- Rationalise and reuse components to build an optimised, non-redundant, consistent and scalable architecture.
- Take part in code reviews and help improve overall code quality.
- Take part in deployments to test and production environments in a DevOps context.
- Manage component upgrades and the related testing.
- Write and maintain technical documentation.
- Join the team’s Agile ceremonies and use tools such as Jira and Confluence.
Technical environment
- Spark 3, ideally 3.2
- Scala 2, ideally 2.12
- Cloudera Hadoop platform
- YARN, HDFS, Hive, HBase, Kafka
- Kerberos, Ranger
- Jenkins, XL Deploy, XL Release
- Control-M
- Bitbucket/Git, Jira, Confluence
- IntelliJ, VS Code
- ZIO, Cats
- 3 to 6 years of experience in data engineering.
- Proficiency in Spark 3 (ideally 3.2) and Spark application optimisation techniques.
- Proficiency in Scala 2 (ideally 2.12) used with Spark, and in coding best practices.
- Very good knowledge of the Hadoop ecosystem (Cloudera, YARN, HDFS, Hive, HBase, Kafka).
- Knowledge of security components such as Kerberos and Ranger.
- Experience implementing CI/CD pipelines (e.g. Jenkins, XL Deploy, XL Release).
- Experience with Control-M scheduling and the Atlassian suite (Bitbucket/Git, Jira, Confluence).
- Familiarity with IntelliJ or VS Code.
- Knowledge of a functional programming framework such as ZIO or Cats is an advantage.
- Ability to understand technical requirements and to plan and estimate tasks.
- Experience working in Agile (Scrum) with a continuous-improvement mindset.
- English: B2 minimum, C1 preferred.
- Ability to adapt communication to infrastructure teams, developers, business analysts and users.
Candidate-se aqui: