Your AI observability needs your agents' memory, not just their traces
We run our AI coding agents on a fixed token budget, and we judge what it buys by outcomes, not output. A merged pull request is output. A customer problem solved, an incident closed or a question answered for good is an outcome. Spending tokens on outcomes starts with seeing where they go, so we built Tach, short for tachymeter, to measure it. Tach records our agents' traces, and it also collects their memory. Memory is where an agent keeps what it has learnt about us, our conventions, our codebase and our taste, and taste is what turns generic code into code we'd have written ourselves. But when we audited all 802 memories, 93 were workarounds for friction in our own tooling, and they were wasting tokens. They loaded into every session, a third of them pointed at problems already fixed, and the friction behind the rest kept costing retries. If you observe your agents, put their memory in your observability layer next to the traces.
Outcomes, not pull requests
Counting pull requests is the easy way to measure agents, and it rewards the wrong thing. An agent can open ten pull requests that move nothing and still look productive. What we want to know is what a session cost, what it delivered, and whether the same outcome could have cost less. That's what we mean by development tokenomics, a word the industry now has a foundation for, and it's the same idea as the cost feedback loop we use to steer agents in the cloud, turned on our own development.
That accounting needs instruments, and Tach is the tachymeter for our development loops and our software factory, built on PostHog with a custom layer on top. PostHog stores the session events: model calls, tool calls, tokens, and the work each session belongs to. Our layer stores and analyses what PostHog wasn't built for, such as memories, and runs higher-level analysis on all of it with agents we fully control.
Memory steers every session
A session doesn't start empty. Before the agent reads the task, Claude Code loads its instruction files and the index of its memory, MEMORY.md. Only the first 200 lines or 25 KB of that index load, whichever comes first, and anything past that point is left out. The memories themselves sit in topic files that the agent opens only when an index line looks relevant to the task. On my machine, the index is 108 lines and costs 7.9k tokens at the start of every session, whatever the task.
Agents like Claude Code manage this memory on their own. When something surprises them, they write a note and add a line to the index, so the next session doesn't pay to learn it again. When the note is right, that's a real saving. But a memory also creates behaviour. A line that says "never run X, run Y instead" changes what every later session does, and nothing prompts anyone to review it.
They're not good at the upkeep yet. They add notes freely and rarely go back to check whether one is still true. Our own harness runs a hook that sends each developer's memories to Tach on a regular schedule, which is how we could look at all of them at once. If your team uses Claude Code or any agent with persistent memory, take a look at yours.
93 memories were workarounds, and workarounds waste tokens
Tach keeps the memories of every developer's agents in one place, so we can manage and maintain them like any other part of the codebase. Our audit had a model sort each memory by what it's for, with the evidence next to it, and we checked a sample of its calls by hand. Most were knowledge worth keeping, but 93 of the 802 were workarounds. In each one, an agent got past a problem and told only itself how.
| Where the friction was | Workaround memories | Share |
|---|---|---|
| Developer tooling (scripts, hooks, CLIs, local stack) | 57 | 61% |
| Data layer | 14 | 15% |
| Infrastructure | 11 | 12% |
| Product | 11 | 12% |
Most of the friction sits in developer tooling, where agents spend their day. Our previous post covered seven of these, about a third-party tool. Here's one of ours. We wrote a guard script so that agents never commit with --no-verify. It read flags from the whole shell line, so git log -n 5 && git commit -m 'fix' was rejected as a --no-verify commit, because of the -n that belonged to git log. The agent's memory recorded the fix that worked for it, run the commit as its own command, and moved on.
The memory told us where to look, and Tach's traces let us put a number on it. In a 30-day sample of 264,714 agent tool calls, the guard rejected a command 149 times, and 144 of those rejections were false alarms. They hit 108 sessions across five of our six developers, and 21 of those sessions hit it more than once. The failed calls and their retries cost about 147,000 output tokens. Each retry also re-reads the session's context. At the size of a fresh session on my machine, 65k tokens, we estimate another 9 million input tokens, mostly prompt-cache reads billed at a tenth of the normal price. All of it bought nothing, because the commit was fine the first time.
A workaround like this gets paid for four times. The agent pays once to find the detour, and every later session pays to load it. The friction stays in place for the next person or agent without that memory. And the line takes room in an index capped at 200 lines, room a useful memory could have had. Proper engineering means finding the root cause and fixing it for good, and a workaround memory hides the root cause from everyone except the agent that wrote it.
We tried to put a token figure on each workaround. Tach sees every file an agent opens, and in the same 30 days our agents opened a workaround memory file only six times. Reading them costs next to nothing, so the waste sits in the index line that loads every session, and in the friction the workaround leaves in place. On my machine, the 31 index lines that point at workarounds cost about 1,200 tokens per session, 15% of the index. My agents ran 304 sessions in the sample, so about 370,000 tokens went to loading workarounds on one machine alone. The friction is harder to price memory by memory, because only some failures leave a signature in the tool calls, the way the guard's rejection message does.
Context rot nobody was monitoring
When we checked the 93 against today's code, a third of the underlying problems had already been fixed, by a person or by an agent working on something else. None of those fixes touched the memory. Tian Pan calls this the moment a fix becomes a memory-invalidation event, and in our data nobody had invalidated anything. On my machine, seven index lines pointed at problems that no longer existed. One told every session that an environment variable breaks four tests in a pre-push hook, and to drop it before pushing. The tests had been fixed weeks before, and every session since had loaded the warning anyway.
A stale memory is the most misleading line in the index. It costs tokens in every session like any other line, but it grounds the agent in a codebase that no longer exists, steering it away from a path that now works. It takes index room from a memory that's still true, and whatever sits past the cap never loads. It also spreads, because a later agent can read it as a rule and copy it into a shared skill.
Anthropic's consolidate-memory skill, available in the Claude app, merges duplicates, prunes stale entries and tightens the index, so we ran it once on my memories. It retired or merged 6 of 192 files, edited 9 and repaired 23 broken links. It checked notes about finished projects against the repo and Linear, and it retired one workaround after checking that its bugs were gone from the code. The other six workarounds for problems already fixed stayed, including the pre-push warning. In that run, the skill verified the notes that describe work, and kept the notes that tell the agent how to behave as rules. Some tools already go further: GitHub Copilot Memory stores a citation with each memory and checks it against the current branch before using it.
It also only runs when someone remembers to run it, on one machine at a time, and it can't see the rest of the team. Our most common lesson was saved on its own by the agents of four of our six developers. A lesson that four machines load belongs on all six, and private memory is the wrong place for it. Once a memory is shared knowledge, it should move into the project's context, the instruction files every agent loads, and leave the private memories it came from.
Monitoring the steering
Tach has collected agent memories from the start, but we built that feature to answer a positive question: which memories should a developer share with the rest of the team? Making local memories converge into a team memory is still the right goal, and the audit showed it isn't enough, because memory creates behaviour, good or bad, and that behaviour is what we have to monitor. So that's the loop we're building into Tach. Every new or changed memory gets checked against the code and Linear. A workaround points to a root cause, and the root cause gets a ticket. Knowledge that several people's agents reached on their own becomes a shared rule or an automated check, and a person approves each change. When a fix merges, the memories that worked around it change with it. Traces show what your agents did. Memory shows what they've been told to do next time. If you observe one, observe the other.