About the role
You will design, implement, and optimize distributed training systems across thousands of GPUs. You will develop advanced parallelization, improve training stability and resource utilization, and build monitoring, visualization, and debugging tools for large-scale training runs.
Responsibilities
- Design, implement, and optimize distributed training systems across thousands of GPUs
- Research and implement FSDP, Tensor Parallel, Pipeline Parallel, and Expert Parallel
- Build monitoring, visualization, and debugging tools for training runs
- Optimize training stability, convergence, and resource utilization across large clusters
Requirements
- Extensive distributed PyTorch training and parallelism experience for foundation-model training
- Deep understanding of GPU clusters, networking, and storage systems
- Familiarity with NCCL, MPI, and distributed-system optimization
Requirements
- Extensive distributed PyTorch training and parallelism experience for foundation-model training
- Deep understanding of GPU clusters, networking, and storage systems
- Familiarity with NCCL, MPI, and distributed-system optimization
Skills
Interested in this role?
You'll apply directly on Luma AI's careers site.