About the role
# Data Platform Engineer
We are looking for a skilled **Senior Data Engineer – Platform & Operations** to manage and improve enterprise data platforms, with responsibilities spanning data engineering, pipeline development, production support and data operations.
The role involves building and maintaining data pipelines, managing large-scale data processing workflows, monitoring platform performance and resolving data-related issues. You will also work closely with business stakeholders and technical teams to understand requirements and deliver reliable data solutions.
The ideal candidate should have strong hands-on experience with Python, PySpark, SQL and Big Data technologies, along with an understanding of production environments, workflow orchestration, data quality and application operations.
## Key Responsibilities
### 1. Data Engineering & Pipeline Development
Required Technical Skills
- Experience:** 4–10 Years (Relevant Experience)
- Location:** Bangalore
- Work Mode:** Hybrid
- Employment Type:** Full-Time
We are looking for a skilled **Senior Data Engineer – Platform & Operations** to manage and improve enterprise data platforms, with responsibilities spanning data engineering, pipeline development, production support and data operations.
The role involves building and maintaining data pipelines, managing large-scale data processing workflows, monitoring platform performance and resolving data-related issues. You will also work closely with business stakeholders and technical teams to understand requirements and deliver reliable data solutions.
The ideal candidate should have strong hands-on experience with Python, PySpark, SQL and Big Data technologies, along with an understanding of production environments, workflow orchestration, data quality and application operations.
## Key Responsibilities
### 1. Data Engineering & Pipeline Development
- Build, maintain and enhance scalable data pipelines using Python, PySpark, SQL and Apache Spark.
- Develop data processing solutions for both batch and streaming workloads.
- Work with large datasets using Hadoop, Hive, HDFS and Spark-based technologies.
- Handle data extraction, transformation, integration and processing across multiple sources.
- Implement data models and transformation logic to support business and reporting requirements.
- Identify performance bottlenecks and optimize existing data pipelines.
- Investigate data inconsistencies, processing failures and technical defects.
- Contribute to platform enhancements, automation and data architecture improvements.
- Monitor scheduled data workflows, applications, microservices and reporting activities.
- Manage data pipeline execution, scheduling and workflow dependencies using Apache Airflow.
- Investigate failed jobs, delayed data processing and application-related issues.
- Perform controlled data reloads, job restarts and reprocessing activities whenever required.
- Monitor data availability, freshness and overall platform performance.
- Support production deployments, releases and ongoing platform maintenance.
- Ensure smooth functioning of data processing systems and minimize operational disruptions.
- Manage production incidents, service requests and change activities following ITIL practices.
- Investigate recurring technical issues and perform root-cause analysis.
- Coordinate with application, infrastructure, development and database teams for issue resolution.
- Work with ITSM platforms such as ServiceNow to track and manage operational tickets.
- Maintain technical documents, troubleshooting guides and operational knowledge resources.
- Provide timely updates to stakeholders regarding production issues and their resolution.
- Perform data validation and quality checks across different processing stages.
- Identify missing records, incorrect values, inconsistencies and data processing errors.
- Conduct unit, integration, regression and functional testing for data solutions.
- Validate data transformations, reporting outputs and downstream data consumption.
- Develop mechanisms to improve data accuracy, reliability and monitoring.
- Ensure that data pipelines meet expected quality and performance standards.
- Develop and maintain reports and dashboards using Power BI.
- Monitor scheduled report refreshes and troubleshoot data connectivity or refresh failures.
- Work with business users to understand reporting requirements and data needs.
- Ensure accurate and consistent data availability for business reporting.
- Collaborate with business users, data engineers, architects and application teams to understand technical requirements.
- Convert business expectations into technical specifications and data engineering solutions.
- Participate in requirement-gathering sessions and technical discussions.
- Evaluate the impact of new requirements on existing data pipelines and platform architecture.
- Identify opportunities to leverage available data for additional business use cases.
- Coordinate with cross-functional teams to ensure smooth implementation of requirements.
- Communicate technical challenges, dependencies and proposed solutions effectively.
Required Technical Skills
- Strong programming skills in Python, PySpark and SQL.
- Hands-on experience with Hadoop, Hive, HDFS, Apache Spark and Spark SQL.
- Good understanding of ETL processes, data transformation, data modelling and pipeline optimization.
- Experience with batch processing and Spark Structured Streaming.
- Working knowledge of MongoDB and relational databases.
- Experience working with Parquet and Avro file formats.
- Hands-on exposure to Apache Kafka and Kafka-based data processing.
- Experience with Apache Airflow for workflow scheduling, monitoring and troubleshooting.
- Knowledge of Power BI for dashboard development and report refresh management.
- Understanding of CI/CD pipelines and deployment tools such as GitHub, Bitbucket or equivalent.
- Working knowledge of Linux / RHEL operating systems.
- Understanding of ITIL processes, incident management and ServiceNow/ITSM tools.
- Familiarity with development tools such as VS Code, Jupyter Notebook or equivalent.