About the role
The Data Engineer designs and builds the real-time data ingestion pipeline from REPLAN's AVEVA PI historian and DeltaV DCS to the AWS cloud layer. This role owns the Edge Connector Agent, AWS Kinesis/Glue ETL pipeline, BadInput detection logic, and the sensor data quality framework.
Data Ingestion Pipeline
Responsibilities:
• Develop the Edge Connector Agent — Python service polling AVEVA PI Web API, transmitting to AWS IoT Core
• Configure AWS IoT Core rules — route sensor messages to Kinesis Data Streams
• Build AWS Glue Streaming ETL job — consume from Kinesis, validate, detect BadInputs, write to S3 and Aurora MySQL
• Implement BadInput detection logic — all 6 check types (zero, out-of-range, stuck, spike, missing, cross-inconsistency)
• Build sensor TAG registry management — engineering range configuration, BAD INPUT rules per TAG
• Design and implement the S3 data lake structure — raw, processed, and results zones with lifecycle policies
Inspection & Static Data Pipelines
• Build inspection data ingestion pipeline — Lambda validation and transformation for NDT results
• Implement presigned S3 URL workflow for large file uploads (3D laser scan point cloud data)
• Build 3D laser scan processing Lambda — trigger FEA geometry update job on scan upload
• Implement S3 lifecycle policies — archive sensor_readings older than 12 months to Parquet in S3
Data Quality & Monitoring
• Build CloudWatch dashboard for data ingestion health — Kinesis lag, Glue job success rate, BadInput rate per asset
• Implement data quality alerting — notify on ingestion failures, AVEVA PI connectivity loss, BadInput rate anomalies
• Build sensor TAG coverage report — identify assets with insufficient sensor coverage for FEA boundary conditions
• Maintain AVEVA PI tag registry — update as PETROBRAS adds or modifies instrumentation
Mandatory Requirements
• Bachelor's in Computer Science, Data Engineering, or equivalent
• 5+ years data engineering experience
• Strong Python skills — experience building data pipeline services
• AWS data services: Kinesis Data Streams, AWS Glue (Streaming ETL), AWS Lambda, S3, IoT Core
• Experience with AVEVA PI Web API or OSIsoft PI SDK for process historian integration
• Experience with Apache Kafka, AWS Kinesis, or equivalent streaming platforms
• SQL expertise — Aurora MySQL optimisation and time-series query patterns
• Experience with Docker containerisation for edge applications
Preferred / Advantageous
• Experience with OT/IT integration (OPC-UA, DeltaV, DCS historian systems)
• Knowledge of industrial sensor data patterns and anomaly detection
• Experience with AWS Glue DataBrew or Apache Spark for data quality
• Familiarity with PETROBRAS AVEVA PI or similar historian environments
Key Performance Indicators (KPIs)
• Edge Connector Agent deployed and data flowing from REPLAN to AWS within Phase 1 timeline
• BadInput detection pipeline processing 100% of sensor readings with < 1-second latency from ingestion
• Data quality: < 2% data loss rate from ingestion to processed zone under normal conditions
• Glue ETL job success rate ≥ 99.5% per month
• Sensor TAG registry complete and validated for all 15 assets before Phase 1 FEA pilot
What do we offer?
- TCS Benefits – Brazil:
- Health insurance
- Dental Plan
- Life insurance
- Transportation vouchers
- Meal/Food Voucher
- Childcare assistance
- TotalPass
- TCS Cares – free 0800 that provides psychological assistance (24 hrs/day), legal, social and financial assistance to associates
- Partnership with SESC
- Reimbursement of Certifications
- Free TCS Learning Portal – Online courses and live training
- International experience opportunity
- Discount Partnership with Universities and Language Schools
- Bring Your Buddy – By referring people you become eligible to receive a bonus for each hire
- TCS Gems – Recognition for performance
- Xcelerate – Free Mentoring Career Platform
At TATA Consultancy Services we promote an inclusive culture, we always work for equity. This applies to Gender, People with Disabilities, LGBTQIA+, Religion, Race, Ethnicity. All our opportunities are based on these principles. We think of different actions of inclusion and social responsibility, in order to build a TCS that respects individuality. Come to be a TCSer!