Skip to main content
1 line of config → $2,200 a year: 100 Lambda workers had the throughput of 10

1 line of config → $2,200 a year: 100 Lambda workers had the throughput of 10

· 6 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

1 line of deploy configuration → $2,200 a year.

That's what one AWS Lambda function saves by running 15 copies instead of 100. Every night it works through about 34,000 queued messages, and the batch ends at the same time whether 40 copies run or 100. The database behind it is saturated at a median of 4.7 runs a second, so the extra copies just wait their turn.

On serverless, that waiting is billed, because Lambda charges for the time a function runs. Each run takes 17 seconds instead of 2, and the bill pays for the difference. Nothing in the code or its tests could show it. This is the third post of Feedback loops from the FinOps trenches, after a streaming pipeline that turned a data lake bill into an API bill and a CI scan whose bill nobody saw.

Runs finished per second in each hour of the nightly batch, by the average number of copies running. The points sit around 4.7 runs per second from 40 copies to 100. A dashed model line rises from zero and reaches that ceiling at about 10 copies

Throughput shows the ceiling​

Run time is the first thing anyone sees in this function's metrics. When few copies run, the median run takes 2.1 seconds, and when about 100 run, it takes 17. That points to too many copies without saying how many is right, because the quiet hours are the start and the end of the batch, when the queue is almost empty.

Throughput answers it. We took 276 hours of the nightly batch's metrics and computed, for each hour, the runs finished per second and the average number of copies running. The second comes from Little's law, which says that average copies running equals runs per second times seconds per run.

The chart above shows the result. From 40 copies to 100, throughput stays flat around 4.7 runs a second. Every copy past that point queued in the database, whose CPU peaks at 86% on a typical night and has reached 98%.

Where the knee is​

If one run takes 2.1 seconds when nothing is in its way, and the database completes 4.7 runs a second, then about 4.7 × 2.1 ≈ 10 copies keep it busy. That's the knee. Below it, more copies make the batch faster, and above it, they wait.

The ceiling moves with the other load on the database. In 11 of the 241 busy hours it reached 7 to 10 runs a second, which would put the knee as high as 20 copies. So the answer is a range, settled by measuring. Set the limit around 15, watch runs per second for a few nights, and step it down while throughput holds. Above the knee, the batch takes about two hours whatever the setting, because 34,000 runs at 4.7 a second take that long. Any team, or any agent, can repeat the method on the next function.

Why the cost stayed hidden​

Setting 100 was a reasonable call. More workers is the usual answer to "make the batch faster", and on a laptop or in a test environment it works. The ceiling only exists at runtime, in the production environment, with production volume and every other client of the same database competing for it. The team that wrote the function had no environment where that signal could appear before the deploy.

After the deploy, the signal stayed quiet. The function reported no errors, the queue drained, and the batch ended before morning. A run time that swings between 2 and 17 seconds reads as normal variance and pages nobody. The bill shows Lambda compute at about $225 a month, a total with no hint that the workers spend most of it waiting on each other.

The pattern travels. Any team that turns up parallelism to go faster can walk into the same ceiling, and on a bigger function the same curve costs much more.

For a function fed by a queue, AWS offers a second knob, the event source's maximum concurrency, which caps how many copies one queue can start. It's independent of reserved concurrency, and AWS warns that if reserved concurrency is set lower than the queues' maximum, Lambda may throttle messages. Pick one ceiling and keep the other above it.

The loop a software factory needs​

This is the cost feedback loop at work. Three runtime numbers line up in the same hour: runs per second stop rising, run time keeps rising, and the database runs near its limit. Together they say that more workers buy queueing here, and they give the right setting directly, as the ceiling times the uncontended run time. The change is two characters, from 100 to 15, for about $2,200 a year, more than $1,000 a year per character typed.

A software factory can run that test after every deploy, like a unit test: did this change add copies without adding throughput? The change it points to is one number in the function's deploy configuration, and the next nights' throughput says whether it worked and whether the limit can go lower. Run after the deploy that raised the limit, the check flags the ceiling on the first night.

This is the kind of loop Cloud ex Machina lets a software factory run, with a price attached. It reads the runtime metrics and the bill every day, puts a dollar figure on the waste, and measures the next bills after the change, so the factory learns what each fix was worth.

We track these signals whether or not a factory is listening. Cloud ex Machina turns them into application-layer FinOps recommendations, each with its estimated saving and the team that owns the code, and the line to change once the repository is connected.

Before you raise concurrency​

  • Measure runs per second as well as run time. Flat throughput means you've found a ceiling.
  • Find what each copy waits on. If it's a database, its capacity sets the knee.
  • Compute the knee before you pick a number: ceiling × uncontended run time. Then confirm it with a few nights of data.