AI Infrastructure Engineer, Model Serving Platform

Position Overview

As a Software Engineer on the ML Infrastructure team, you will design and build platforms for scalable, reliable, and efficient serving of LLMs. Our platform powers cutting-edge research and production systems, supporting both internal and external use cases. The ideal candidate combines strong ML fundamentals with deep expertise in backend system design. You’ll work in a highly collaborative environment, bridging research and engineering to deliver seamless experiences to our customers and accelerate innovation across the company.

Key Responsibilities

• Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale.
• Build an internal platform to empower LLM capability discovery.
• Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.
• Conduct architecture and design reviews to uphold best practices in system design and scalability.
• Develop monitoring and observability solutions to ensure system health and performance.
• Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment.

Required Qualifications

• 4+ years of experience building large-scale, high-performance backend systems.
• Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++).
• Experience with LLM serving and routing fundamentals (e.g., rate limiting, token streaming, load balancing, budgets, etc.).
• Experience with LLM capabilities and concepts such as reasoning, tool calling, and prompt templates.
• Experience with containers and orchestration tools (e.g., Docker, Kubernetes).
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).
• Proven ability to solve complex problems and work independently in fast-moving environments.

Preferred Qualifications

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

Benefits & Perks

• Compensation: Base salary range of $175,000—$220,000 USD, including equity and additional compensation components.
• Benefits: Comprehensive health, dental, and vision coverage; retirement benefits; a learning and development stipend; generous PTO; and eligibility for additional benefits such as a commuter stipend.