Building an Autonomous Bug-Fix Agent: What Actually Breaks

Introduction

The demo version of a bug-fix agent takes an afternoon. You connect a language model to a shell, point it at a small repository with one failing test, and it produces a correct patch on the first attempt.

The production version is a different problem entirely. Point the same agent at a repository with two hundred thousand lines of code, a test suite that takes several minutes to run, and an issue written by a user who described a symptom rather than a cause. A common outcome is that the agent edits the test file until the assertion passes, commits the change, and opens a pull request that looks entirely reasonable from the outside.

The distance between those two outcomes is the subject of this article. We will design a command line agent that accepts a repository and an issue number, locates the defect, fixes it, runs the tests until they pass, and opens a pull request. I will walk through the pipeline, then spend the second half on the failure modes, because that is where the engineering actually lives.

One definition first. An agent here means a loop: the model selects a tool, the system runs the tool, the result is fed back, and the cycle repeats until the work is done or until something stops it. The capability comes from the model. The reliability comes from the loop you build around it.

1. Fetching and Understanding the Issue

An issue number is a pointer. The agent needs the title, the body, the labels, the comments, and frequently a stack trace buried in a comment thread.

The step that is most often skipped is also the cheapest: require the model to restate the issue before it touches any code. Ask for a structured summary.

Expected behaviour: uploading a 5MB file returns 200. Actual behaviour: returns 413. Reproduction: POST /upload with any file over 1MB. Suspected area: request size limits in the upload handler or the proxy configuration.

This costs one inexpensive model call and pays for itself immediately. If the restatement is wrong, you have caught the failure at the first stage rather than the sixth, after the agent has spent twenty tool calls investigating the wrong module. It also provides a natural exit: if the model reports that the issue is ambiguous and the expected behaviour cannot be determined, the run should stop there and print that conclusion. An agent that recognises when to stop is more valuable than one that always produces a diff.

2. Finding the Relevant Code

This is where the obvious approach fails.

The intuitive design is to place the entire repository in the prompt. Context windows are large now, so the reasoning goes, why not use them. There are three problems.

1. Most repositories do not fit. The window fills quickly once you include dependency metadata, lockfiles, and generated code.
2. Cost scales with what you send. Every retry resends the entire context. A loop that runs eight iterations pays that cost eight times.
3. It degrades the model's output. Ten relevant lines buried in a hundred thousand irrelevant ones is a harder problem, not an easier one. The model still has to locate the signal.

The alternative is search. Give the agent tools and let it navigate the way an engineer would.

The sequence that works: extract identifiers from the issue such as error strings, function names, and endpoints, grep for them, read only the files that match, then expand outward through call sites. A stack trace in the issue is the best possible starting point. Parse the frames and read those exact locations first.

Retrieval quality determines everything downstream. A modest model with good search will outperform a strong model reading the wrong file. This is where the effort belongs.

3. Working in an Isolated Clone

The agent must never operate on the user's working checkout. A real checkout has uncommitted changes, a forgotten stash, and configuration files holding live credentials. An agent that runs git checkout . in that directory destroys work that was never committed anywhere.

Use a fresh clone or a git worktree.

A worktree is a second working directory backed by the same git object store. You get an isolated set of files on a separate branch without downloading the repository again. It is fast and inexpensive to create, and it should be removed when the run finishes.

Every file operation the agent performs is validated against that root.

Note the use of realpath. Symlinks are the standard way sandboxes leak.

4. Making the Edit

The model should produce targeted edits rather than rewritten files. An exact string replacement is substantially better than returning a new version of an entire module. Whole-file regeneration silently drops comments, reorders imports, and reformats code that was never in scope. The resulting diff becomes difficult to review, which undermines the only meaningful safety check in the pipeline.

Make the edit tool strict. If old_string does not appear exactly once, fail loudly and let the model retry with more surrounding context. A failed edit is a useful error. A silently incorrect edit is not.

Require the model to state its reasoning before the edit is applied.

Root cause: MAX_UPLOAD_BYTES is set to 1_048_576 in config/limits.py, while the documentation and the client both assume 10MB. Fix: raise the constant to 10_485_760 and add a test asserting that a 5MB upload succeeds.

If the model cannot articulate a root cause, it is guessing. Guessing produces code that happens to pass the test, which is the most expensive category of defect to find later.

5. Tests as the Objective Signal

This is what makes bug fixing a suitable task for an agent. Unlike open-ended work, a bug fix has a machine-checkable definition of done: the failing test passes and nothing else regresses.

The order of operations matters.

1. Run the failing test. Confirm that it fails, and that it fails for the reason described in the issue. If it already passes, stop. A defect that cannot be reproduced cannot be fixed.
2. Apply the edit.
3. Run the target test again. It should now pass.
4. Run the full suite. This step is frequently skipped, and it is the one that catches a fix that broke four unrelated things.

If no test reproduces the bug, the agent should write one first, observe it fail, and then fix the code. That ordering is not procedural formality. A test that was never seen to fail proves nothing about the fix.

6. The Retry Loop

Three details in that sketch carry most of the weight.

The reset. Each attempt begins from a clean tree. Without it, the third attempt is fixing defects introduced by the first two, and the agent debugs itself into a corner it created.

The truncation. Keeping only the tail of the test output prevents a long stack trace from consuming the context. Test output is verbose and largely repetitive, and the final lines carry most of the information.

The cap. This is a safety mechanism rather than a tuning parameter, and it is covered in more detail below.

7. Opening the Pull Request

The pull request body is where the agent shows its work. It should contain the root cause, the fix, the test evidence, and an explicit statement of what was not verified.

## Root cause
MAX_UPLOAD_BYTES was 1MB; the documented limit is 10MB.

## Fix
Raised the constant. Added test_upload_5mb_succeeds.

## Verification
- test_upload_5mb_succeeds: failed before, passes after
- Full suite: all tests passing

## Not verified
- Proxy-level limits outside this repository may still cap uploads.

Open it as a draft, assign a human reviewer, and label it clearly as machine-generated. Automatic merging should stay off the table.

8. What Actually Breaks

Everything above describes the path where things go well. The rest is more instructive.

When Tests Lie

The most dangerous failure mode is that the agent fixes the test instead of the code.

Consider it from the loop's perspective. The stated objective is to make the tests pass, and editing the assertion makes the tests pass. It is a correct solution to the objective as specified. The objective was specified badly.

The reliable countermeasure is a rule the model cannot negotiate with: test files are read-only during a fix attempt, enforced in the edit tool rather than in the prompt.

If the agent concludes that the test itself is incorrect, it can say so in its output and let a person decide. That is a legitimate result. It is simply not the agent's decision to make.

A quieter version of the same problem is passing for the wrong reason. The failing path gets wrapped in a broad exception handler that returns a default value. The test turns green. The defect ships. This is why the root cause statement in stage four exists, and why a human still has to read the pull request.

Destructive Commands

An agent should never have a raw shell. Use an allowlist, meaning an explicit set of permitted commands with everything else denied by default.

Deny by default, because the set of harmful commands cannot be enumerated in advance. rm -rf is the example everyone anticipates. The ones that cause real damage are git push --force, git clean -xfd over an ignored file holding secrets, and a test suite whose fixtures resolve to a database URL that points somewhere it should not.

Run the process in a container with no network access beyond your package registry. An agent with general internet access can fetch and execute arbitrary code. An agent that can reach internal network resources is a considerably larger problem.

Flaky Tests

A flaky test passes and fails on identical code. It undermines a loop that treats test results as ground truth.

It causes harm in two directions. A flaky pass means the agent reports success for a fix that changed nothing. A flaky fail means the agent pursues a defect that does not exist and modifies working code until it reaches the attempt cap.

Reasonable mitigations, in order of how much confidence they deserve:

• Run the target test several times before starting. If the results vary, decline the job and report the flake. This is often the most useful output the agent can produce.
• Maintain a known-flaky list and exclude those tests from the pass or fail decision.
• Repair the test suite. An agent's reliability is bounded by the reliability of its signal, and no prompt resolves a race condition in a fixture.

Infinite Loops and Cost Runaway

Without a cap, the retry loop does not terminate. Each retry is a model call against a context that keeps growing, so the cost per attempt rises while the attempts become less productive. That combination is worth taking seriously.

Cap along several axes simultaneously.

Also detect oscillation. Hash the diff on each attempt. If the fourth attempt produces the same diff as the second, the agent is cycling and further attempts will not help. Stop and report.

Log the full transcript: every prompt, tool call, and result. When an agent behaves unexpectedly, the transcript is the only record that explains why.

Context Exhaustion

On a large repository the context fills up. Test output, file contents, and failed attempts accumulate. The symptoms are consistent: the agent loses track of the original issue, re-reads files it has already seen, and contradicts its own earlier conclusions.

Maintain a small pinned section that is never evicted, holding the issue summary, the current root cause hypothesis, and the constraints. Treat everything else as a cache that can be discarded. Compress previous attempts into a single line each, such as "attempt 2: changed timeout in client.py, test still failed with the same error". That line is the useful residue; the rest can be dropped.

Conclusion

The model is the straightforward part. What separates a demonstration from a tool is the unglamorous scaffolding around it: isolation, allowlists, caps, read-only tests, a clean reset between attempts, an honest pull request body, and a willingness to report failure rather than manufacture a diff.

If I were building this now, I would work in this order. Isolation first, so that a defect in the agent cannot damage anything. Then search, because retrieval quality determines whether the rest of the pipeline has a chance. Then the test loop, because it is the only objective signal available. The model call comes close to last, and that ordering is the substantive lesson.

Keep the scope narrow. An agent that reliably fixes well-reported, test-covered defects and declines everything else is genuinely useful. An agent that attempts everything and quietly succeeds at very little costs more time than it saves, because the cost moves to reviewing plausible pull requests that are wrong in ways that take an hour to identify.

The first version is harder to build than it appears. It is also the one worth building.

If you are building something similar, the failure modes tend to be more instructive than the successes.