AI Evaluation Engineer
You will build automated evaluation benchmarks, hallucination detection suites, and quality scoring systems for client AI models.
What you'll work with.
As an AI Evaluation Engineer at TheWebVale, you will ensure our AI applications are accurate, consistent, and safe before they reach client users. You will construct automated evaluation datasets, grading rubrics, and regression test suites for healthcare, legal, and coaching tools. You will collaborate with AI engineers and client subject matter experts in our two-to-three week release cadences. You will prevent prompt regressions, detect hallucinations, and provide data-backed confidence scores for every AI release.
Your day to day.
6 things, one at a time. Keep scrolling.
Design comprehensive evaluation datasets representing edge cases and common user prompts.
Build automated testing harnesses that grade model outputs using heuristic and LLM judges.
Track accuracy, factual consistency, relevance, and safety metrics across prompt iterations.
Implement automated regression testing in CI pipelines before deploying prompt changes.
Analyze model failures and collaborate with prompt engineers to refine system instructions.
Generate readable evaluation reports showing accuracy improvements over project iterations.
What we look for.
6 things, one at a time. Keep scrolling.
Hold 2–4 years of software engineering or data analysis experience with AI testing focus.
Demonstrate high proficiency in Python, Pandas, regular expressions, and API testing.
Show practical experience with evaluation frameworks like Ragas, DeepEval, or Promptfoo.
Understand statistical sampling, inter-annotator agreement, and qualitative grading rubrics.
Possess analytical rigor with zero tolerance for undetected model hallucinations.
Communicate evaluation outcomes and reliability risks objectively to project stakeholders.
What you get.
Three steps, no tricks.
- 1
Intro chat
A short call about what you want to work on and how you like to work.
- 2
Work review
Walk us through projects, code or designs you are proud of.
- 3
Paid task
A small paid task or pairing session, so both sides see the fit.
Apply for AI Evaluation Engineer .
Tell us about your work. We reply to every application, usually within a few days.