{"title":"Exercise: Agent Evaluation and Testing","description":null,"content":"# Exercise 1: Build an Agent Eval Harness\n\nMeasure an agent instead of eyeballing it: define a set of tasks with known correct answers, run the agent on each, check the result, and report a success rate.\n\nYou cannot improve what you do not measure. An agent that looks good on a handful of manual tries often fails in ways you only catch with a repeatable test set. An eval harness gives you a single number, success rate, that you can track as you change prompts, tools, or models, so improvements and regressions are visible.\n\n### Requirements\n\n- Define an eval set: tasks paired with a way to check the answer (for example, an expected substring).\n- Run the agent (a model call) on each task.\n- Check each output against its expected answer and record pass or fail.\n- Print per-task results and the overall success rate.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\n# (task, expected_substring) the answer should contain\neval_set = [\n (\"What is the capital of France? Answer with one word.\", \"Paris\"),\n (\"What is 15 + 27? Answer with the number only.\", \"42\"),\n (\"What language is the Django framework written in? One word.\", \"Python\"),\n]\n\ndef run_agent(task: str) -> str:\n # TODO: send the task to the model and return its answer\n pass\n\n# TODO: run the agent on each task, check whether the expected substring is in\n# the answer (case-insensitively), print pass/fail per task, and print the\n# overall success rate\n```\n\n\n### Expected Output\n\nEach task reports pass or fail, and the harness prints a single success rate.\n\n\n```plaintext\nPASS | What is the capital of France? Answer with one word.\nPASS | What is 15 + 27? Answer with the number only.\nPASS | What language is the Django framework written in? One word.\n\nSuccess rate: 3/3 (100%)\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI()\n\neval_set = [\n (\"What is the capital of France? Answer with one word.\", \"Paris\"),\n (\"What is 15 + 27? Answer with the number only.\", \"42\"),\n (\"What language is the Django framework written in? One word.\", \"Python\"),\n]\n\ndef run_agent(task: str) -> str:\n response = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"user\", \"content\": task}],\n )\n return response.choices[0].message.content.strip()\n\npassed = 0\nfor task, expected in eval_set:\n answer = run_agent(task)\n ok = expected.lower() in answer.lower()\n passed += ok\n print(f\"{'PASS' if ok else 'FAIL'} | {task}\")\n\nrate = passed / len(eval_set)\nprint(f\"\\nSuccess rate: {passed}/{len(eval_set)} ({rate:.0%})\")\n```\n\n\nThe harness turns judgment into measurement: every task carries its own check, so the success rate is computed rather than estimated. A case-insensitive substring check is a simple but effective grader for short factual answers, and you would reach for an LLM-as-judge when answers are open-ended. The value is repeatability. Run the same set after changing a prompt or model, and the change in success rate tells you whether you helped or hurt.\n\n---\n\n# Exercise 2: Grade Open-Ended Answers with an LLM Judge\n\nA substring check only works when answers are short and exact. For open-ended answers, you need a judge: a second model that compares the answer to a reference and decides pass or fail. This is how you evaluate explanations, summaries, and reasoning where many wordings are correct.\n\n### Requirements\n\n- Keep an eval set of questions paired with reference answers.\n- Run the agent to produce an answer for each question.\n- Write a `judge` that asks a model to grade the answer against the reference, returning PASS or FAIL.\n- Print per-question results and the overall pass rate.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\n# (question, reference_answer) for open-ended grading\neval_set = [\n (\"Explain what a cache is in one sentence.\", \"A cache stores data so future requests are served faster.\"),\n (\"What problem does pagination solve?\", \"It splits large result sets into smaller pages to avoid huge responses.\"),\n]\n\ndef run_agent(question):\n response = client.chat.completions.create(model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"user\", \"content\": question}])\n return response.choices[0].message.content.strip()\n\ndef judge(question, reference, answer):\n # TODO: ask a judge model whether `answer` is correct given the reference.\n # Return True for pass, False for fail.\n ...\n\n# TODO: run the agent on each question, judge each answer, and print pass/fail + the pass rate\n```\n\n\n### Expected Output\n\nEach answer is graded by the judge against its reference, and the harness reports a pass rate.\n\n\n```plaintext\nPASS | Explain what a cache is in one sentence.\nPASS | What problem does pagination solve?\n\nPass rate: 2/2 (100%)\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport json\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\neval_set = [\n (\"Explain what a cache is in one sentence.\", \"A cache stores data so future requests are served faster.\"),\n (\"What problem does pagination solve?\", \"It splits large result sets into smaller pages to avoid huge responses.\"),\n]\n\ndef run_agent(question):\n response = client.chat.completions.create(model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"user\", \"content\": question}])\n return response.choices[0].message.content.strip()\n\ndef judge(question, reference, answer):\n response = client.chat.completions.create(\n model=\"gpt-5.5\",\n messages=[{\"role\": \"system\", \"content\": \"You grade answers. Reply with only PASS or FAIL. PASS if the answer is consistent with the reference.\"},\n {\"role\": \"user\", \"content\": f\"Question: {question}\\nReference: {reference}\\nAnswer: {answer}\"}],\n )\n return response.choices[0].message.content.strip().upper().startswith(\"PASS\")\n\npassed = 0\nfor question, reference in eval_set:\n answer = run_agent(question)\n ok = judge(question, reference, answer)\n passed += ok\n print(f\"{'PASS' if ok else 'FAIL'} | {question}\")\n\nprint(f\"\\nPass rate: {passed}/{len(eval_set)} ({passed / len(eval_set):.0%})\")\n```\n\n\nAn LLM judge handles the variability a substring check cannot: two correct explanations can share almost no words, so you grade against meaning, not text. Constraining the judge to reply PASS or FAIL keeps its output parseable and the pass rate computable. The judge is not infallible, so for high-stakes evals you would calibrate it against human labels, but it scales grading of open-ended output far past what manual review can cover.","pageType":"ai-engineering"}

Get Premium