Skip to main content
26 lines of YAML → $4,700 a year saved on a CI scan nobody saw the bill for

26 lines of YAML → $4,700 a year saved on a CI scan nobody saw the bill for

· 9 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

26 lines of YAML → $4,700 a year saved.

That's what one GitHub Actions security scan was worth once it checked only the code each merge changed. Before the fix, it cost about $2 every time someone merged to the main branch. Nobody looks twice at $2, but at 48 merges a week it made up roughly three quarters of the repository's CI bill, mostly spent re-checking code that had been checked minutes earlier.

Now the same check costs $126 a year instead of $4,830. The fix was one pull request. The hard part was getting the cost in front of the people who write CI, a group that now includes coding agents, because none of them ever see the bill.

1 line of config → $2,200 a year: 100 Lambda workers had the throughput of 10

1 line of config → $2,200 a year: 100 Lambda workers had the throughput of 10

· 6 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

1 line of deploy configuration → $2,200 a year.

That's what one AWS Lambda function saves by running 15 copies instead of 100. Every night it works through about 34,000 queued messages, and the batch ends at the same time whether 40 copies run or 100. The database behind it is saturated at a median of 4.7 runs a second, so the extra copies just wait their turn.

On serverless, that waiting is billed, because Lambda charges for the time a function runs. Each run takes 17 seconds instead of 2, and the bill pays for the difference. Nothing in the code or its tests could show it. This is the third post of Feedback loops from the FinOps trenches, after a streaming pipeline that turned a data lake bill into an API bill and a CI scan whose bill nobody saw.

A streaming pipeline turned a data lake bill into an API bill

A streaming pipeline turned a data lake bill into an API bill

· 6 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

A data lake bucket held less than a terabyte and cost about $5 a day. On the day two new Databricks streaming pipelines went live, its cost rose to almost $19 a day, and S3 request costs made up 97% of it. The pipelines worked as written, the tables stayed fresh, and every dashboard stayed green. Two configuration defaults drove the extra cost, and the signal sat on the cloud bill, far from the pipeline code and its tests. A few lines of configuration fixed it, for about $5,000 a year. This post opens Feedback loops from the FinOps trenches, our series on runtime signals that point back at the code that caused them.

The tokenomics of software development: checking AI-written code now costs as much as writing it

The tokenomics of software development: checking AI-written code now costs as much as writing it

· 13 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

Our AI coding agents now spend about as much checking and shipping code as writing it. We got there in six steps past the review wall, the point where agents write code faster than people can review it. Each step was paid for in tokens, and each one handed us the next problem. Here they are, with our own numbers.

A third of our agents' workaround memories outlived the bug

A third of our agents' workaround memories outlived the bug

· 8 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

We audited all 802 memories our coding agents had saved. 93 were workarounds for friction in our own tooling, and a third pointed at problems already fixed. One hid a guard script that raised 144 false alarms in 30 days. Each workaround loads into every session, and the friction behind it keeps costing retries until someone fixes the cause. Memory steers what every session does and spends, so we audited it next to the traces.

Our agents filed seven bug reports. They sent them to themselves.

Our agents filed seven bug reports. They sent them to themselves.

· 8 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

In October we read all 802 notes our coding agents had saved for themselves. Seven described rtk quietly mangling their output. Each one ended with the same workaround, and each sat in one developer's private memory. Upstream rtk had shipped the fix on September 11. Reading the notes turned seven workarounds into one upgrade, on October 28.

Your AI writes code blind to cloud cost. Here's how to close the loop.

Your AI writes code blind to cloud cost. Here's how to close the loop.

· 12 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

AI coding agents write a growing share of production code, and none of them see what it costs to run. A few services at a US fintech dropped a single AppConfig token and paid about $20k a year for it, while their code passed every test.

Engineers care about this. Nothing in the workflow tells them, though. The agent that wrote the change never finds out it doubled the reads on a database, or that a config it touched now bills a full delivery on every poll. So the next change gets written the same way, just as blind.