AI Evaluation Engineer

Own design, development, and maintenance of evaluation frameworks and benchmark suites for AI/LLM-powered features, focusing on accuracy, safety, and compliance.

Implement automated evaluation pipelines integrated into CI/CD to detect regressions before release and track quality metrics over time.

Lead investigation and root-cause analysis of failures across model and product boundaries, ensuring trustworthy delivery of AI features handling regulated securities and compliance data.

5+ years in QA/test engineering, ML evaluation, or related quality-focused engineering role.

Hands-on experience testing or evaluating AI/LLM features including building evaluation sets and scoring rubrics.

Proficiency in scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration.

Experience working directly with LLMs or AI tooling (prompt engineering, RAG pipelines) as a developer or evaluator.

Experienced in blending AI model evaluation with broader product quality and compliance testing in regulated environments.

Analytical mindset to quantitatively define and defend metrics for subjective AI quality questions and ensure auditability.

Skilled in cross-functional collaboration with product managers and engineers to translate ambiguous quality requirements into measurable criteria.