Skip to main content
Our agents filed seven bug reports. They sent them to themselves.

Our agents filed seven bug reports. They sent them to themselves.

· 8 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

In October we read all 802 notes our coding agents had saved for themselves. Seven described rtk quietly mangling their output. Each one ended with the same workaround, and each sat in one developer's private memory. Upstream rtk had shipped the fix on September 11. Reading the notes turned seven workarounds into one upgrade, on October 28.

Why we run rtk on every shell command​

AI coding agents live in the shell, ours included. Every line of output they read lands in the model's context and stays there until the session ends or gets compacted. Most of it is noise, like the hundreds of lines a passing test suite prints when the agent needs one.

rtk is an open-source proxy, written in Rust, that cuts that noise. It is one of the tools we'd least like to give up. A Claude Code hook rewrites git diff into rtk git diff. rtk runs the real command and returns a version filtered for that tool. Test runs come back as their failures, git output comes back compacted, and the agent has nothing extra to do.

On the author's laptop alone, rtk has kept 6.7 million tokens out of agent context, 13.8% of everything the commands it handled printed.

Compression is the right trade when the agent only reads the output. It breaks when the output feeds a patch file, a count or another agent.

Seven agent memories, one workaround​

Our agents keep private notes between sessions, so that a lesson learned on Monday is still there on Tuesday. When we triaged all 802 of them, seven came from the agents of two developers and described rtk failures.

DateWhat the agent recorded
Aug 12gh pr edit and gh pr comment lose their subcommand, so gh prints its usage instead
Aug 28Search flags get mangled, and a commit reports success but creates nothing
Sep 18git diff > patch.diff writes a summary instead of a patch, so git apply refuses it
Sep 23A search returns zero hits for a pattern that matches
Sep 28git log --format='%s' > file drops most of the subjects
Sep 28gh pr diff N > file drops about half of the diff
Sep 29A lint run reports exit 2 although the lint passed

Each memory ends by telling the agent to run the command through rtk proxy, which skips the filter. Each agent found the bug, found a way around it and wrote both down. The note went to a memory file that only that developer's agents load.

One of these failures reached a release note. Our deployment agent writes the note from the commit subjects of the release. For the October 7 release, the first version of the note listed fewer than half of the pull requests that shipped. A person spotted the gap and fixed it 44 minutes after the deploy. The code that shipped was right, and only its internal description was wrong. No command had failed, so nothing raised a warning.

The workaround also turned into a habit. In a sample of 206,965 shell commands we pulled for this post, 7,128 went through rtk proxy. Agents of five of our six developers use it. In the same sample, only 33 of 5,952 failed shell commands mention rtk. Truncated output looks like a success, so it rarely shows up among the failures.

xkcd 1172, Workflow: an update fixes the CPU heating up when the space bar is held, and a user complains that this was their workflow

xkcd 1172: Workflow by Randall Munroe, CC BY-NC 2.5.

Reproducing it​

A memory is only the agent's own account, so on October 28 we re-ran the commands on the setup the agents had, rtk 0.37.2 and our hook script.

Command, output written to a fileThrough the hookUnfiltered
gh pr diff on a 25-file PR23,379 chars, 0 file headers58,840 chars, 25 file headers
git log --format='%s' over a long commit range1% of the subjectsAll of them
git diff over 20 commits580 lines, 0 file headers2,485 lines, 62 file headers

Two memories didn't reproduce, the empty searches and a push failure whose cause went missing, so we left them out.

Upstream had already fixed it​

Timeline: upstream rtk fixed redirected output on September 11, five of seven agent memories came after that, and we upgraded on October 28

We were on rtk 0.37.2, released April 20. The latest release when we checked was 1.1.0, from October 25. Current rtk ships a rtk rewrite command that decides which commands are safe to rewrite. Upgrading the binary doesn't replace a hook that's already installed, and our hook dated from February, so it never called rtk rewrite.

On 1.1.0, rtk rewrite leaves alone every command we tested whose output goes to a file or into wc or head. The changelog puts that work in 0.49.0, on September 11, and five of the seven memories are from after that date.

One rough edge remained in 1.1.0 when we checked. When an agent reads gh pr diff directly, it sees 14 of the 25 files. The output ends with ... (more changes truncated) and a hint to run rtk git diff --no-compact. That hint names the wrong command. In a clean checkout it prints nothing, so an agent that follows it concludes there's nothing more to see. rtk gh pr diff --no-compact prints all 25 files. The same wrong hint on git show is open as rtk#4817, and both commands share the code that prints it. Related open issues cover silent loss on large diffs, swallowed exit codes and rewritten pipe producers.

Why coding agents don't report bugs​

rtk had bugs. Its maintainers fixed most of them in 0.49.0, on September 11. For 47 days, the fix sat in a public release and the evidence that we needed it sat in our agents' memory files. Reading the memories connected the two.

A developer who hits the same tool failure twice gets annoyed. Annoyance is what makes people check for an update or file an issue. An agent doesn't get annoyed. Its job ends with the task, so it writes down the workaround and moves on. The workaround works, so the note stays a private instruction. Seven agent sessions had found the bug, and reading their notes brought it to a person.

What we're changing​

Anything a machine reads, like a patch or a diff we hand to a review agent, now goes through rtk proxy or a script the hook can't reach.

We upgraded to rtk 1.1.0 and regenerated the hook with rtk init -g. The git log that kept 1% of the subjects now keeps all of them. We also pinned rtk, so it now gets updated with our other dependencies, hook included. It sees every shell command our agents run, so it gets the same care as anything in our lockfile.

The bigger change is that a workaround in a memory now counts as a bug report. The weekly memory review we're building will send each one to a ticket rather than to an instruction file. rtk was the easy case, because its maintainers had already done the fixing and all we had to do was upgrade. Across all 802 memories, 93 (12%) record a workaround for a defect. Most of those defects live in our own code and tools, where the fix is ours to make.

The loop behind a workaround​

A workaround in an agent's memory is a runtime signal. It points back at the tool or the code that caused it. Closing the loop takes three things: the signal goes back to that tool or code, it reaches an owner, and a check says whether the fix worked. For rtk, the check was the git log row in the reproduction table: 1% of the subjects before the upgrade, all of them after.

A software factory is a set of agents that write a change, test it and ship it. Its agents meet these signals every day and, left alone, keep them in private memory. Cloud ex Machina runs and prices this loop for software factories: it ties a runtime signal to the tool or code behind it, gives it an owner and a check, and measures the result after the fix.

Check your own agents' memory​

Search your agents' memory files for "proxy", "instead" or "workaround". Each hit is a workaround that has not reached its owner yet. If you run rtk, re-run rtk init -g after every upgrade, because upgrading the binary leaves an installed hook as it was.