About the role
As an AI Evaluation Engineer at TheWebVale, you will ensure our AI applications are accurate, consistent, and safe before they reach client users. You will construct automated evaluation datasets, grading rubrics, and regression test suites for healthcare, legal, and coaching tools.
You will collaborate with AI engineers and client subject matter experts in our two-to-three week release cadences. You will prevent prompt regressions, detect hallucinations, and provide data-backed confidence scores for every AI release.
What you will do
- Design comprehensive evaluation datasets representing edge cases and common user prompts.
- Build automated testing harnesses that grade model outputs using heuristic and LLM judges.
- Track accuracy, factual consistency, relevance, and safety metrics across prompt iterations.
- Implement automated regression testing in CI pipelines before deploying prompt changes.
- Analyze model failures and collaborate with prompt engineers to refine system instructions.
- Generate readable evaluation reports showing accuracy improvements over project iterations.
What you will need
- Hold 2–4 years of software engineering or data analysis experience with AI testing focus.
- Demonstrate high proficiency in Python, Pandas, regular expressions, and API testing.
- Show practical experience with evaluation frameworks like Ragas, DeepEval, or Promptfoo.
- Understand statistical sampling, inter-annotator agreement, and qualitative grading rubrics.
- Possess analytical rigor with zero tolerance for undetected model hallucinations.
- Communicate evaluation outcomes and reliability risks objectively to project stakeholders.
Nice to have
- Experience setting up LLM-as-a-judge pipelines with calibrated scoring rubrics.
- Familiarity with security testing against prompt injection and jailbreak vulnerabilities.
- Knowledge of structured synthetic test data generation using language models.