Measure an agent instead of eyeballing it: define a set of tasks with known correct answers, run the agent on each, check the result, and report a success rate.

You cannot improve what you do not measure. An agent that looks good on a handful of manual tries often fails in ways you only catch with a repeatable test set. An eval harness gives you a single number, success rate, that you can track as you change prompts, tools, or models, so improvements and regressions are visible.

Requirements

Expected Output

Each task reports pass or fail, and the harness prints a single success rate.

Solution

The harness turns judgment into measurement: every task carries its own check, so the success rate is computed rather than estimated. A case-insensitive substring check is a simple but effective grader for short factual answers, and you would reach for an LLM-as-judge when answers are open-ended. The value is repeatability. Run the same set after changing a prompt or model, and the change in success rate tells you whether you helped or hurt.

A substring check only works when answers are short and exact. For open-ended answers, you need a judge: a second model that compares the answer to a reference and decides pass or fail. This is how you evaluate explanations, summaries, and reasoning where many wordings are correct.

Requirements

Expected Output

Each answer is graded by the judge against its reference, and the harness reports a pass rate.

Solution

An LLM judge handles the variability a substring check cannot: two correct explanations can share almost no words, so you grade against meaning, not text. Constraining the judge to reply PASS or FAIL keeps its output parseable and the pass rate computable. The judge is not infallible, so for high-stakes evals you would calibrate it against human labels, but it scales grading of open-ended output far past what manual review can cover.