About the role
Job Description :
Build and scale data pipelines for NyaayOS, processing large volumes of messy legal documents. Work with Python, AWS, PostgreSQL, OCR and async workflows to build reliable ingestion, document processing, data-quality and provenance systems.
What we're looking for :
- 4+ yrs in data/backend engineering.
- Strong Python, AWS, PostgreSQL & production data pipelines.
- Hands-on OCR/document processing and async workflows using Airflow, Celery, Dagster or Temporal.
Key Responsibilities :
- Design and implement scalable ETL pipelines to ingest and transform complex, unstructured data sets into structured formats for downstream analytics.
- Develop and maintain high-performance data processing workflows using Airflow to ensure timely and reliable data availability.
- Integrate OCR technologies into existing data pipelines to automate the extraction of information from varied document formats.
- Optimize database performance and schema design within PostgreSQL to support high-concurrency read and write operations.
- Build and manage distributed task queues using Celery and Temporal to handle asynchronous processing and fault-tolerant background jobs.
- Deploy and manage cloud-native data infrastructure on AWS, ensuring cost-efficiency and high availability of data services.
- Collaborate with engineering leadership to define data architecture standards and improve overall system reliability.