Match Score
Against your primary resumeLogin to See Your Match Score
Create a free account or log in to unlock your CV match score across:
Protocol Intelligence
Data-driven signals on your job's competitivenessLog in to see why each signal reads the way it does.
Job Description
Structured overview of role & requirementsAbout This Role
Design, build, and own evaluation infrastructure including CI/CD pipelines and datasets to measure the quality of AI agents in production.
Diagnose failures in AI agent pipelines and develop prototypes or prioritize fixes to improve reliability and efficiency.
Lead and manage the AI Quality engineering team, guiding technical direction while contributing hands-on to code and experimentation.
Minimum Requirements
7+ years experience building and shipping production software, preferably with complex LLM-powered agents using multiple tools or sub-agents.
Proficiency or readiness to learn Ruby on Rails and Python; Ruby on Rails used in production system.
Experience building evaluation or observability infrastructure for ML/AI systems, including eval pipelines or dashboards.
Experience designing datasets and annotation workflows for ML/AI evaluation.
Ideal Candidate Profile
Experienced in complex AI agent systems focused on action-taking and multi-turn conversational agents rather than simple assistants.
Able to lead a technical team in a fast-paced, ambiguous environment with frequent experimentation and iterative learning.
Comfortable partnering closely with AI Core engineering teams to translate measurements into actionable improvements and model decisions.