In short
Senior (14+ yrs) Data Architect role designing open-source enterprise data platforms — Spark, Kafka, Airflow, Trino, lakehouse tables (Iceberg/Hudi/Delta) — plus AI/ML platform architecture. Day to day: HLD/LLD, cloud migrations, DataOps governance, and client-facing solutioning in a BFSI context.
About the role
- Skills Required:
- 14+ years of experience in Enterprise Data Architecture, Open-Source Data Platforms, Distributed Data Processing, and Data Engineering.
- Proven experience in Enterprise Solution architecture using Open source technology stack across the Data to Insights end to end value chain.
- Strong hands-on expertise in Apache Spark, PySpark, Kafka, Flink, Airflow, NiFi, Trino, SQL, Python, Scala, and Java.
- Proven experience in designing and implementing open-source data platform architectures, including component selection, integration, scalability, interoperability, and platform modernization.
- Deep understanding of batch, streaming, and real-time data processing architectures, including performance tuning, workload optimization, and failure recovery.
- Experience building modern Data Lakehouse architectures using Apache Iceberg, Hudi, Delta Lake, object storage, and open table formats.
- Strong knowledge of event-driven architectures, CDC, APIs, schema management, and enterprise integration patterns.
- Expertise in DataOps practices, including Airflow orchestration, CI/CD, automated testing, observability, metadata management, lineage, and platform operations.
- Experience with PostgreSQL, Cassandra, MongoDB, OpenSearch, and enterprise analytics platforms, including data modeling, indexing, workload management, and availability design.
- Strong understanding of security, governance, and compliance, including RBAC, identity management, encryption, privacy controls, auditability, metadata governance, and vulnerability management.
- Proven ability to lead architecture definition, HLD/LLD creation, NFR assessment, migration planning, technical governance, and production readiness activities.
- Experience supporting large-scale data platform modernization, cloud migration, solutioning, RFP responses, effort estimation, and client-facing architecture discussions.
- Good understanding of BFSI regulatory, security, resilience, privacy, and operational requirements.
- Bachelor’s degree in computer science, Engineering, Information Systems, Data Management, or a related discipline.
- Preferred certifications in Cloud Data Engineering, Kubernetes, Enterprise Architecture, Data Management, Security, DevOps, or Open-Source Data Platforms.
- Experience assessing business use cases and selecting the appropriate approach across BI/KPI reporting, rules, statistical methods, forecasting, machine learning, and GenAI, with clear baselines, acceptance criteria, and measurable business KPIs.
- Understanding of AI/ML platform architecture supporting feature engineering, model training, batch and real-time inference, and integration of ML and GenAI workloads with enterprise open-source data platforms.
- Understanding of MLOps and LLMOps capabilities, including experiment tracking, model registry, feature stores, pipeline orchestration, automated quality gates, and versioning of data, models, and prompts using tools such as MLflow and Kubeflow.
- Exposure to AI observability and monitoring for data drift, model drift, LLM quality, tracing, latency, availability, alerting, and operational runbooks.