Exercise: Coding Agents
Exercise 1: Generate, Run, and Fix Code
Build the loop at the core of a coding agent: ask the model to write a function, run it against a test, and if it fails, feed the error back so the model can fix it. Repeat until the test passes or you run out of attempts.
A coding agent is not magic; it is a generate-execute-observe loop. The model writes code, your harness runs it, and the result (a pass or a traceback) becomes the next input. The execution step is what grounds the agent: instead of trusting that the code looks right, you find out whether it actually works and let the model react to real feedback.
Requirements
- Ask the model to write a Python function for a given task.
- Run the generated code against a simple test in a subprocess with a timeout.
- If it fails, send the error back and ask for a fix.
- Loop up to a few attempts; print each attempt's outcome and the final working code.
Expected Output
The agent writes code, the harness runs it, and the loop ends when the test passes.
Solution
The loop closes the gap between generated code and working code. Each iteration runs the model's output in a subprocess and turns the result into feedback: a pass ends the loop, a failure becomes a new message containing the exact error. That error is the signal the model needs to correct itself, which is why agents that execute and re-prompt outperform single-shot generation. The timeout and three-attempt cap are non-negotiable in practice, since executing untrusted code demands real sandboxing and a bound on retries.
Exercise 2: Pass a Multi-Function Test Suite
A real coding task has more than one requirement and a test suite that checks all of them. Ask the agent to write two functions, run them against a suite of assertions, and let it self-correct on failure until every assertion passes. More requirements mean more chances to fail, which makes the execute-and-fix loop earn its keep.
Requirements
- Ask for two functions in one task, with a test suite covering both.
- Run the generated code against the suite in a subprocess.
- On failure, feed the error back and retry, up to three attempts. Print each outcome and the final code.
Expected Output
The agent writes both functions and passes the suite, usually on the first or second attempt.
Solution
A multi-assertion suite is a stricter grader: the code must satisfy every requirement, not just look plausible, and a single failing assertion sends the exact error back for correction. This is closer to how coding agents work against real test suites, where the tests define done. The timeout and attempt cap remain essential, since running model-written code needs sandboxing and a bound on retries.