In short
Builds, scales, and operates a petabyte-scale data lakehouse on AWS behind spatial data products: ingesting and versioning robotics/headset/video captures, running distributed CPU/GPU processing, and feeding GPU clusters for 3D reconstruction (SLAM, Gaussian Splats) and ML training. Core stack includes Spark, Ray, Flink, Iceberg, LakeFS, and Terraform.
About the role
Our client is looking for a hands-on Senior Data Infrastructure Engineer to build, scale, and operate the petabyte-scale data lakehouse behind our spatial data products.
Every field and lab capture—robotics, egocentric headsets, multi-camera video, depth, IMU—lands in this platform, gets versioned, and powers downstream 3D reconstructions (Gaussian Splats, SLAM) and ML training clusters.
This is a true engineering role. We need someone who has personally built and run a PB-scale platform in production, not just used one.
Key Responsibilities
- PB-Scale Lakehouse: Design object storage layouts (S3), partitioning strategies, catalog architectures, and open table formats (Iceberg/Delta).
- Data Versioning & Ingest: Build reproducible dataset lineage (LakeFS) and high-throughput ingest pipelines (validation, checksumming, deduplication, bronze $→$ gold promotion).
- Distributed CPU/GPU Compute: Operate large-scale processing (Spark, Ray, Flink) for frame extraction, sync/calibration, SLAM/3DGS batch jobs, and auto-labeling.
- ML Training Integration: Keep GPU clusters fed using sharded training formats (WebDataset, Mosaic Streaming), caching, and optimized cross-region transfers.
- Curation & Querying: Build indexing, search, and high-performance SQL query layers (Trino, DuckDB, Spark SQL) so researchers can slice data without copying it.
- Reliability & Infrastructure: Maintain IaC (Terraform) on AWS, handle pipeline backfills/recovery, optimize storage/egress costs, and participate in on-call rotation.
Qualifications
- Experience: 6+ years in data infrastructure or distributed systems engineering.
- Scale: Proven track record setting up and operating a live petabyte-scale data platform (able to speak concretely to throughput, object limits, and failure modes).
- Languages: Python + at least one of Go, Rust, C++, Java, or Scala.
- Table Formats & Versioning: Iceberg/Delta Lake, Catalogs (Glue/Nessie), LakeFS or content-addressed storage.
- Processing & Orchestration: Spark, Ray, or Flink; Airflow, Argo, or Prefect.
- Cloud/IaC: Strong AWS (S3, EKS/ECS, IAM) and Terraform experience.
Nice to Have
- Experience with unstructured/multimodal spatial data (video, LiDAR, point clouds, ROS bags/MCAP).
- GPU-accelerated pipelines (NVIDIA DALI, NVDEC, RAPIDS) or ML data loaders (WebDataset, Petastorm).
- Query engines (Trino, DuckDB, ClickHouse) and embedding/vector search.