Senior QA Engineer, AI Agents & Agentic Workflows

Own continuous validation and quality assurance of a multi-agent AI product with ~20 evolving agents, focusing on their reasoning, tool use, orchestration, and response quality.

Design and maintain end-to-end business scenario tests and evaluation frameworks that handle non-deterministic, LLM-driven systems, including automated regression suites and eval datasets.

Identify and triage multi-agent orchestration and integration issues, test for hallucinations, unsafe responses, and quantify regressions due to model or prompt changes, collaborating closely with Engineering and Product teams.

Bachelor's degree in software engineering, computer science, or related field.

3-5 years of QA experience, including 1-2 years specifically testing LLM-based applications, AI agents, or agentic workflows.

Experience designing evaluation frameworks for non-deterministic AI outputs, including building AI evaluation or regression suites (e.g., LLM-as-judge, golden datasets).

Hands-on familiarity with agentic AI concepts (tool/function calling, multi-agent orchestration, planning, RAG) and test automation tools (e.g., Selenium, Playwright).

Experienced QA professional skilled in testing complex AI agent systems with multi-step workflows and non-deterministic outputs.

Proficient in both human and automated evaluation methods to judge correctness, consistency, and safety of AI agent responses across evolving models and prompts.

Able to analyze and troubleshoot root causes in multi-agent orchestration environments, differentiating issues related to prompt, model, retrieval, tool, or orchestration components.