Senior/Staff AI Cloud Infra Engineer

Position Overview

We are looking for an AI Cloud Infra Engineer to join our infrastructure team. This role is responsible for ensuring the reliability of our back-end systems, collaborating with fellow engineers, and planning for future growth. Our core infrastructure relies heavily on Kubernetes (K8s), Terraform, and GCP, but we value your ability to learn, adapt, and ship robust solutions above the specifics of any tool.

Key Responsibilities

• Develop and maintain our core Python platform for routing requests, orchestrating AI workloads, managing GPU server capacity, observability, and more.
• Develop and maintain our infrastructure layer using Terraform and cloud provider APIs to manage our fleet of GPU workers across cloud and potentially bare metal environments.
• Own and operate the technologies underpinning our platform, including K8s, FluxCD, Nomad, Prometheus, Thanos, DataDog, Loki, and distributed networking/storage.
• Architect and implement solutions that directly impact the performance and availability of services for millions of ComfyUI users.
• Work closely with the core engineering team to design and build new infrastructure systems.
• Help create the vision and lay the foundation for the future direction of our infrastructure over the next 1/2/5 years.
• Contribute to shaping our technical direction and infrastructure best practices as we grow.

Required Qualifications

• Relevant experience as an AI Cloud Infra Engineer in a high tech startup environment.
• Experience participating in incident management processes.
• Strong foundation in managing cloud infrastructure (AWS, GCP, or Azure). Experience with bare metal is a plus.
• Solid understanding of container orchestration (Kubernetes preferred) and CI/CD principles and tools.
• Excellent communication skills.
• Proven ability to learn fast and ship quality infrastructure code and configurations.

Preferred Qualifications

• Experience excelling in or a passion for fast-paced, high-growth startup environments.
• Experience with GPU management, scheduling, and monitoring in a large-scale environment.
• Familiarity with observability tools, such as DataDog.

Benefits & Perks

• Compensation Range: $150K - $300K