Gen AI / ML Application Testing Engineer (BA + QA)

Position Overview

Job Title: Gen AI / ML Application Testing Engineer (BA + QA)
Location: San Jose, CA – Hybrid (Tue, Wed, Thu – ONLY LOCALS)
Duration: Long Term

We are looking for a detail-oriented engineer with experience in Gen AI / ML application testing, business analysis, and product validation. You will help shape the quality of next-gen AI products through systematic testing, prompt validation, and tool-driven evaluation.

Key Responsibilities

• Design and execute test cases for Gen AI / ML features and user workflows
• Use GenAI-specific tools to validate prompt consistency, hallucination detection, etc.
• Collaborate with product managers to convert requirements into test cases and test data
• Perform exploratory testing, regression, and prompt-based scenario testing
• Write automation scripts to simulate user behavior and backend interactions
• Track and manage issues using QA platforms and agile tools
• Document test plans, test reports, and AI evaluation metrics

Required Qualifications

• Hands-on testing experience with Gen AI / ML products
• Experience with LLM testing tools such as:
  • Promptfoo (prompt testing & evaluation)
  • LangSmith (LangChain tracing & evals)
  • TruLens (feedback tracking for LLMs)
  • Rebuff (security and behavior testing)
• Solid understanding of LLM behavior, hallucinations, and prompt design
• Scripting in Python, Shell, or JavaScript
• Experience with REST APIs, JSON, and YAML
• Familiarity with PyTest, Postman, Selenium, or similar tools
• Bachelor's or Master's in CS, Data Science, AI/ML, or a related field

Preferred Qualifications

• Experience testing RAG, chatbot, or LLM agent systems
• Familiarity with LangChain, LlamaIndex, or Haystack
• Business analysis experience in AI projects
• Knowledge of AI/ML model evaluation metrics