{"title":"Exercise: Coding Agents","description":null,"content":"# Exercise 1: Generate, Run, and Fix Code\n\nBuild the loop at the core of a coding agent: ask the model to write a function, run it against a test, and if it fails, feed the error back so the model can fix it. Repeat until the test passes or you run out of attempts.\n\nA coding agent is not magic; it is a generate-execute-observe loop. The model writes code, your harness runs it, and the result (a pass or a traceback) becomes the next input. The execution step is what grounds the agent: instead of trusting that the code looks right, you find out whether it actually works and let the model react to real feedback.\n\n### Requirements\n\n- Ask the model to write a Python function for a given task.\n- Run the generated code against a simple test in a subprocess with a timeout.\n- If it fails, send the error back and ask for a fix.\n- Loop up to a few attempts; print each attempt's outcome and the final working code.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport subprocess\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\ntask = \"Write a Python function is_palindrome(s) that returns True if s reads the same forwards and backwards.\"\ntest = \"assert is_palindrome('racecar') and not is_palindrome('hello')\\nprint('PASS')\"\n\nmessages = [\n {\"role\": \"system\", \"content\": \"You write Python. Reply with only the function code, no explanation or markdown fences.\"},\n {\"role\": \"user\", \"content\": task},\n]\n\ndef run_code(code: str) -> tuple:\n program = code + \"\\n\" + test\n result = subprocess.run([\"python\", \"-c\", program], capture_output=True, text=True, timeout=10)\n return result.returncode == 0, (result.stdout + result.stderr).strip()\n\n# TODO: loop up to 3 attempts: get code from the model, run it, and if it fails\n# append the error as feedback and ask for a fix. Print outcomes and final code.\n```\n\n\n### Expected Output\n\nThe agent writes code, the harness runs it, and the loop ends when the test passes.\n\n\n```plaintext\nAttempt 1: PASS\nFinal code:\ndef is_palindrome(s):\n return s == s[::-1]\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport subprocess\n\nclient = OpenAI()\n\ntask = \"Write a Python function is_palindrome(s) that returns True if s reads the same forwards and backwards.\"\ntest = \"assert is_palindrome('racecar') and not is_palindrome('hello')\\nprint('PASS')\"\n\nmessages = [\n {\"role\": \"system\", \"content\": \"You write Python. Reply with only the function code, no explanation or markdown fences.\"},\n {\"role\": \"user\", \"content\": task},\n]\n\ndef run_code(code: str) -> tuple:\n program = code + \"\\n\" + test\n result = subprocess.run([\"python\", \"-c\", program], capture_output=True, text=True, timeout=10)\n return result.returncode == 0, (result.stdout + result.stderr).strip()\n\nfinal_code = \"\"\nfor attempt in range(1, 4):\n response = client.chat.completions.create(model=\"gpt-5.5\", messages=messages)\n code = response.choices[0].message.content\n final_code = code\n\n passed, output = run_code(code)\n print(f\"Attempt {attempt}: {'PASS' if passed else 'FAIL'}\")\n\n if passed:\n break\n\n # Feed the failure back so the model can fix it\n messages.append({\"role\": \"assistant\", \"content\": code})\n messages.append({\n \"role\": \"user\",\n \"content\": f\"That failed with:\\n{output}\\nFix the function. Reply with only the code.\",\n })\n\nprint(\"Final code:\")\nprint(final_code)\n```\n\n\nThe loop closes the gap between generated code and working code. Each iteration runs the model's output in a subprocess and turns the result into feedback: a pass ends the loop, a failure becomes a new message containing the exact error. That error is the signal the model needs to correct itself, which is why agents that execute and re-prompt outperform single-shot generation. The timeout and three-attempt cap are non-negotiable in practice, since executing untrusted code demands real sandboxing and a bound on retries.\n\n---\n\n# Exercise 2: Pass a Multi-Function Test Suite\n\nA real coding task has more than one requirement and a test suite that checks all of them. Ask the agent to write two functions, run them against a suite of assertions, and let it self-correct on failure until every assertion passes. More requirements mean more chances to fail, which makes the execute-and-fix loop earn its keep.\n\n### Requirements\n\n- Ask for two functions in one task, with a test suite covering both.\n- Run the generated code against the suite in a subprocess.\n- On failure, feed the error back and retry, up to three attempts. Print each outcome and the final code.\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport subprocess\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\ntask = \"Write two Python functions: is_even(n) returning True for even ints, and factorial(n) returning n!.\"\ntests = \"assert is_even(4) and not is_even(3)\\nassert factorial(5) == 120 and factorial(0) == 1\\nprint('PASS')\"\n\nmessages = [\n {\"role\": \"system\", \"content\": \"You write Python. Reply with only the code, no markdown fences.\"},\n {\"role\": \"user\", \"content\": task},\n]\n\ndef run_code(code):\n program = code + \"\\n\" + tests\n result = subprocess.run([\"python\", \"-c\", program], capture_output=True, text=True, timeout=10)\n return result.returncode == 0, (result.stdout + result.stderr).strip()\n\n# TODO: loop up to 3 attempts. Generate code, run the test suite, and on failure feed the\n# error back for a fix. Print each attempt outcome and the final code.\n```\n\n\n### Expected Output\n\nThe agent writes both functions and passes the suite, usually on the first or second attempt.\n\n\n```plaintext\nAttempt 1: PASS\nFinal code:\ndef is_even(n):\n return n % 2 == 0\n\ndef factorial(n):\n result = 1\n for i in range(2, n + 1):\n result *= i\n return result\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nfrom openai import OpenAI\nimport subprocess\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\ntask = \"Write two Python functions: is_even(n) returning True for even ints, and factorial(n) returning n!.\"\ntests = \"assert is_even(4) and not is_even(3)\\nassert factorial(5) == 120 and factorial(0) == 1\\nprint('PASS')\"\n\nmessages = [\n {\"role\": \"system\", \"content\": \"You write Python. Reply with only the code, no markdown fences.\"},\n {\"role\": \"user\", \"content\": task},\n]\n\ndef run_code(code):\n program = code + \"\\n\" + tests\n result = subprocess.run([\"python\", \"-c\", program], capture_output=True, text=True, timeout=10)\n return result.returncode == 0, (result.stdout + result.stderr).strip()\n\nfinal_code = \"\"\nfor attempt in range(1, 4):\n code = client.chat.completions.create(model=\"gpt-5.5\", messages=messages).choices[0].message.content\n final_code = code\n passed, output = run_code(code)\n print(f\"Attempt {attempt}: {'PASS' if passed else 'FAIL'}\")\n if passed:\n break\n messages.append({\"role\": \"assistant\", \"content\": code})\n messages.append({\"role\": \"user\", \"content\": f\"The tests failed:\\n{output}\\nFix the code. Reply with only the code.\"})\n\nprint(\"Final code:\")\nprint(final_code)\n```\n\n\nA multi-assertion suite is a stricter grader: the code must satisfy every requirement, not just look plausible, and a single failing assertion sends the exact error back for correction. This is closer to how coding agents work against real test suites, where the tests define done. The timeout and attempt cap remain essential, since running model-written code needs sandboxing and a bound on retries.","pageType":"ai-engineering"}

Get Premium