Software Engineering - Inference Engineer

Position Overview

Location: San Francisco, CA (Onsite | Remote). At Virtue AI—a well-funded, early-stage startup founded by industry veterans—our advanced AI security platform sets the standard in the industry. We are looking for an Inference Engineer to own how models are served in production, ensuring that inferences are fast, stable, observable, and cost-efficient even under unpredictable workloads.

Key Responsibilities

• Serve and optimize inference for LLMs, embedding, and other ML models across multiple families.
• Design and operate inference APIs with clear contracts, versioning, and backward compatibility.
• Build routing and load-balancing logic including multi-model routing and fallback/degradation strategies (using vLLM or SGLang).
• Package inference services into production-ready Docker images.
• Implement logging and metrics for inference systems (e.g., latency, throughput, token counts, GPU utilization, Prometheus-based metrics).
• Analyze server uptime and failure modes (such as GPU OOMs, hangs, slowdowns, fragmentation) and implement recovery and restart strategies.
• Design GPU and model placement strategies, including sharding, replication, batching, and managing tradeoffs between latency, cost, and availability.
• Collaborate with backend, platform (Cloud, DevOps), and ML teams to align inference behavior with product requirements.

Required Qualifications

• Bachelor's degree or higher in Computer Science, Computer Engineering, or a related field.
• Strong experience serving LLMs and embedding models in production environments.
• Hands-on experience designing inference APIs and load balancing/routing logic.
• Familiarity with SGLang, vLLM, TensorRT, or similar inference frameworks.
• Strong understanding of GPU behavior including memory limits, batching, fragmentation, and utilization.
• Proficiency with Docker, Prometheus-based metrics, and structured logging.
• Proven ability to debug and fix inference failures in production, including support for autoscaling services and meeting production SLAs.
• Ability to work effectively in a fast-paced startup environment with high ownership.

Preferred Qualifications

• Expertise in GPU-level optimization including memory planning, reuse, kernel launch efficiency, and reducing fragmentation.
• Experience with kernel or runtime-level optimizations (such as CUDA kernels, Triton kernels, or custom ops).
• Knowledge of model-level inference optimization techniques (e.g., quantization [FP8/INT8/BF16], KV-cache optimization, speculative decoding, or batching strategies).

Benefits & Perks

• Competitive salary + equity
• Direct ownership of inference reliability and performance.
• Engage with hard problems at the intersection of systems, GPUs, and AI.
• Production impact – your work directly affects latency, cost, and uptime.
• Be part of a strong technical culture where engineers debug and optimize, not just prototype.