About the role
Mandatory:
- Basic concepts of Data Engineering, ingestion from diverse sources and file formats, Hadoop, Data Warehousing, designing & implementing large-scale distributed data platforms & data lakes
- Building distributed platforms or services
- SQL, Spark, query tuning & performance optimization
- General advanced Scala experience (e.g. functional programming, using case classes, complex data structures & algorithms)
- Experience on SOLID & DRY principles and good software architecture & design experience
- Languages: Python, Scala
- Good experience in Big Data unit, system, integration & regression testing
- DevOps experience in Jenkins, Maven, GitHub, Artifactory/JFrog, CI/CD
- Big Data processing: Hadoop, Sqoop, Spark and Spark Streaming
- Hadoop distributions: Cloudera / Hortonworks experience
- Data streaming: experience on Kafka and Spark Streaming
- Data validation & data quality
- Data Lake & Medallion Architecture
- Shell scripting & automation using Ansible or related configuration management tools
- Agile processes & tools like Jira & Confluence
- Code management tools like Git
- File formats like ORC, Avro, Parquet, JSON & CSV
- Big Data orchestration: NiFi, Airflow, Spark on Kubernetes, YARN, Oozie, Azkaban
Good to have:
- Building and managing large-scale on-prem Hadoop clusters or on cloud
- Data Catalog & Data Governance
- Customer-facing experience
- Experience with a wide range of Big Data tools, databases & frameworks e.g. Kafka Connect, Apache Arrow, Apache Iceberg, Flink
- Cloud native storage like Ceph or S3
- Cloud Data Engineering (Azure, AWS & GCP)
- SQL & NoSQL DB: Hive, Cassandra, Dremio, Presto
- Infrastructure: hands-on experience on Kubernetes, Docker and related container technologies
- Data security & compliance: experience working on GDPR & regulatory frameworks
- Security & secrets management like Ansible Vault, Snyk, Git Secrets, HashiCorp Vault
What would you do here:
- Should have worked on building large-scale distributed systems
- Should have led a team in a tech lead/module lead role
- Should have good mentorship experience
- Should have good communication and very good documentation skills
- Should show maturity & understand the requirement and convert to high-quality technical requirements
- Should code and design end-to-end data flow and deliver on time
- Be resilient & flexible to work across multiple teams and internal teams
- Should help to implement best practices in Data Architecture and Enterprise Software Development
- Should have extensive experience in working in Agile Data Engineering teams
- Work very closely with Engineering Manager, TPM, Product Manager and stakeholders
- Become a quick learner and be more proactive in understanding the wider business requirements and linking to the various other concepts in the domain
- Help implement better solutions independently and faster with better ownership
- Help automate manual operational tasks and focus on creating reusable assets and propel innovation
- Work very closely with team members and be able to have healthy relationships
- Be innovative and able to come up with ideas and reusable components & frameworks
- Should be ready to support 24x7 as per the rota