{"title":"Exercise: Prompt Optimization","description":null,"content":"# Exercise 1: Measure and Compare Two Prompts\n\nOptimize a prompt by measurement, not by feel: define a small labeled test set, run two candidate prompts against it, score each by accuracy, and pick the winner.\n\nPrompt optimization is an evaluation problem. Without a test set, \"this prompt seems better\" is a guess that often does not survive contact with real inputs. With even a handful of labeled examples, you can put a number on each prompt and make the choice defensible, which is exactly how prompt changes get validated in production.\n\n### Requirements\n\n- Define a labeled test set: a few inputs paired with their correct labels.\n- Write two candidate prompts, for example a terse one and a more specific one.\n- Run each prompt across the test set and compute its accuracy.\n- Print both accuracies and declare the winner.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\n# (input, correct_label)\ntest_set = [\n (\"This movie was a masterpiece.\", \"positive\"),\n (\"Worst purchase I have ever made.\", \"negative\"),\n (\"It arrived on time.\", \"neutral\"),\n (\"I absolutely loved every minute.\", \"positive\"),\n (\"The screen cracked after one day.\", \"negative\"),\n]\n\nprompt_a = \"Classify the sentiment.\"\nprompt_b = \"Classify the sentiment as exactly one of: positive, negative, neutral. Reply with only the label.\"\n\ndef accuracy(system_prompt: str) -> float:\n # TODO: run the prompt over the test set, compare each prediction to the\n # correct label, and return the fraction correct\n pass\n\n# TODO: compute accuracy for prompt_a and prompt_b, print both, and print the winner\n```\n\n\n### Expected Output\n\nThe more specific prompt scores at least as well, and usually better, because its output is easier to match against the labels.\n\n\n```plaintext\nPrompt A accuracy: 0.60\nPrompt B accuracy: 1.00\nWinner: Prompt B\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI()\n\ntest_set = [\n (\"This movie was a masterpiece.\", \"positive\"),\n (\"Worst purchase I have ever made.\", \"negative\"),\n (\"It arrived on time.\", \"neutral\"),\n (\"I absolutely loved every minute.\", \"positive\"),\n (\"The screen cracked after one day.\", \"negative\"),\n]\n\nprompt_a = \"Classify the sentiment.\"\nprompt_b = \"Classify the sentiment as exactly one of: positive, negative, neutral. Reply with only the label.\"\n\ndef accuracy(system_prompt: str) -> float:\n correct = 0\n for text, label in test_set:\n response = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": text},\n ],\n )\n prediction = response.choices[0].message.content.strip().lower().rstrip(\".\")\n if label in prediction:\n correct += 1\n return correct / len(test_set)\n\nscore_a = accuracy(prompt_a)\nscore_b = accuracy(prompt_b)\n\nprint(f\"Prompt A accuracy: {score_a:.2f}\")\nprint(f\"Prompt B accuracy: {score_b:.2f}\")\nprint(f\"Winner: {'Prompt B' if score_b >= score_a else 'Prompt A'}\")\n```\n\n\nThe test set turns prompt comparison into a measurement. Normalizing the prediction (lowercasing and stripping a trailing period) before checking it against the label is what makes a free-text reply gradable, and it is also why the constrained Prompt B tends to win: its output is closer to the label to begin with. The same loop scales to dozens of examples and as many candidate prompts as you want to compare.\n\n---\n\n# Exercise 2: Rank Several Candidate Prompts\n\nComparing two prompts is the warm-up; real optimization searches a set of candidates. Score several prompts on the same labeled test set, rank them by accuracy, and report the winner. This is a one-step prompt search, the basis for automated prompt tuning.\n\n### Requirements\n\n- Keep the labeled test set and define three or more candidate prompts of varying specificity.\n- Reuse the `accuracy` function to score each candidate.\n- Sort the candidates by accuracy, print them ranked, and name the winner.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\n# (input, correct_label)\ntest_set = [\n (\"This movie was a masterpiece.\", \"positive\"),\n (\"Worst purchase I have ever made.\", \"negative\"),\n (\"It arrived on time.\", \"neutral\"),\n (\"I absolutely loved every minute.\", \"positive\"),\n (\"The screen cracked after one day.\", \"negative\"),\n]\n\ncandidates = [\n \"Classify the sentiment.\",\n \"Classify the sentiment as positive, negative, or neutral.\",\n \"Classify the sentiment as exactly one of: positive, negative, neutral. Reply with only the label.\",\n]\n\ndef accuracy(system_prompt):\n # TODO: run the prompt over the test set and return the fraction correct\n ...\n\n# TODO: score every candidate, then print them ranked best-to-worst with the winner\n```\n\n\n### Expected Output\n\nThe most specific, label-constrained prompt usually ranks first; vaguer prompts score lower because their output is harder to match.\n\n\n```plaintext\n1. 1.00 Classify the sentiment as exactly one of: positive, negative, neutral. Reply with only the label.\n2. 0.80 Classify the sentiment as positive, negative, or neutral.\n3. 0.60 Classify the sentiment.\n\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\ntest_set = [\n (\"This movie was a masterpiece.\", \"positive\"),\n (\"Worst purchase I have ever made.\", \"negative\"),\n (\"It arrived on time.\", \"neutral\"),\n (\"I absolutely loved every minute.\", \"positive\"),\n (\"The screen cracked after one day.\", \"negative\"),\n]\n\ncandidates = [\n \"Classify the sentiment.\",\n \"Classify the sentiment as positive, negative, or neutral.\",\n \"Classify the sentiment as exactly one of: positive, negative, neutral. Reply with only the label.\",\n]\n\ndef accuracy(system_prompt):\n correct = 0\n for text, label in test_set:\n response = client.chat.completions.create(\n model=\"gpt-5.4-mini\",\n messages=[{\"role\": \"system\", \"content\": system_prompt}, {\"role\": \"user\", \"content\": text}],\n )\n prediction = response.choices[0].message.content.strip().lower().rstrip(\".\")\n if label in prediction:\n correct += 1\n return correct / len(test_set)\n\nscored = sorted(((accuracy(p), p) for p in candidates), reverse=True)\nfor rank, (score, prompt) in enumerate(scored, start=1):\n print(f\"{rank}. {score:.2f} {prompt}\")\nprint(\"Winner:\", scored[0][1])\n```\n\n\nScoring every candidate on the same fixed test set turns prompt choice into a ranking problem with a number behind each option, so the winner is defensible rather than a hunch. Sorting by accuracy surfaces it directly. This is exactly the loop an automated prompt optimizer runs at scale: propose candidate prompts, evaluate each against held-out labels, and keep the best, which is why a reliable `accuracy` function and a representative test set matter more than any single clever prompt.","pageType":"ai-engineering"}

Get Premium