About the role
Runpod, the AI Developer Cloud, is seeking an ML Systems Engineer focused on inference. You will own end-to-end LLM serving performance, measure throughput and latency, and drive optimizations across models and hardware generations.
This is a hands-on role dedicated to reducing cost per token and improving reliability in production. You’ll work with vLLM and SGLang, optimize for API-level latency, and collaborate with product and infra teams to craft scalable, production-grade runtimes and
Define how to measure inference performance (throughput, time to first token, inter-token latency, cost per token). Profile and diagnose performance problems across the serving stack. Improve serving efficiency for large models on single-node and multi-node GPUs. Turn findings into production-ready runtimes, configurations, and defaults. Collaborate with product and infra teams to shape inference offerings. Keep up with the inference ecosystem and decide what to adopt or contribute back. Trace bottlenecks and implement fixes when tuning alone isn’t enough. 5+ years of professional system engineering experience. Hands-on experience with vLLM, SGLang in production or benchmark scale. Strong software engineering skills in Python; performance-critical codebases. Understanding of batching, memory, parallelism, latency vs throughput trade-offs. Experience with inference optimization techniques (quantization, speculative decoding, distributed serving). Proficiency in benchmarking and GPU profiling tools. Equity Medical/Dental/Vision Flexible PTO Remote work first Home office stipend