Skip to main content
A streaming pipeline turned a data lake bill into an API bill

A streaming pipeline turned a data lake bill into an API bill

· 6 min read
Samuel Cozannet
Co-founder & CPTO, Cloud ex Machina

A data lake bucket held less than a terabyte and cost about $5 a day. On the day two new Databricks streaming pipelines went live, its cost rose to almost $19 a day, and S3 request costs made up 97% of it. The pipelines worked as written, the tables stayed fresh, and every dashboard stayed green. Two configuration defaults drove the extra cost, and the signal sat on the cloud bill, far from the pipeline code and its tests. A few lines of configuration fixed it, for about $5,000 a year. This post opens Feedback loops from the FinOps trenches, our series on runtime signals that point back at the code that caused them.

Daily list and write requests to one data lake bucket, in millions. About 0.8 million a day before two streaming pipelines went live, then 2 to 5 million a day after, with write requests up more than tenfold

A very diligent pipeline​

The two pipelines are Lakeflow Declarative Pipelines on Databricks, formerly Delta Live Tables, running in continuous mode, so they "process new data as it arrives" and never stop. Their source changes rarely. Every 35 to 70 seconds, each pipeline checked for new data, found none, and wrote down that it had found none. One of the two tables had never received a row.

Each of those notes costs requests. A streaming job records its progress in a streaming checkpoint, and it splits its state into one folder per shuffle partition. Spark's spark.sql.shuffle.partitions defaults to 200, and the checkpoint we looked at had 201 state folders. Each empty micro-batch listed and wrote files across all of them.

In the 32 days after go-live, the bucket averaged:

Per dayBeforeAfterChange
List requests0.76 M2.49 M×3.3
Write requests0.07 M0.99 M×13.6
Bucket cost$5.20$18.88×3.6

S3 bills lists and writes at the same $0.005 per thousand, more than twelve times the price of a read. The bucket's own bill shows it: 76.2 million list requests cost $380.98 in one month, while storage stayed under $1 a day.

The bucket also has versioning turned on. When the job cleaned up an old checkpoint file, S3 kept the file as a noncurrent version and added a delete marker. The bucket gained 60 million objects in a month, mostly checkpoint leftovers that nobody would ever read.

Why the cost stayed hidden​

Continuous mode is a sensible choice when you want fresh tables, and 200 partitions is what Spark gives you unless you ask for another number. The people who built the pipelines made reasonable choices with the information they had. That information came from the code and its tests, and the cost of these two choices only appears at runtime, in the production environment, at production cadence, after days of empty batches.

Nothing in their daily view carried that signal. Tests and staging runs last minutes. The pipeline's own metrics showed fast, successful batches. The cloud bill groups the charge as "S3 requests" for a bucket that many jobs share, a line that no single team owns, and it reaches finance as a monthly total weeks after the change that caused it.

That gap is the subject of our earlier post on AI-generated code and cloud cost: whoever writes the change never sees what it costs to run.

The fix fits in a pipeline config​

The pipelines moved to triggered mode on a schedule that matches how often their data changes, with a handful of shuffle partitions instead of 200. The checkpoint fixes the partition count, so a new count needs a fresh checkpoint, which costs nothing for a table with no rows. A lifecycle rule now expires noncurrent versions after a few days.

A cloud team would usually find the lifecycle rule first, and on its own it saves about $4 a month. Almost all of the $5,000 a year sits in the pipeline configuration, which a data engineer owns. A cost tool that stops at the bucket settings misses it.

The loop a software factory needs​

The number we keep coming back to is the ratio: a few lines of configuration for about $5,000 a year, more than $1,000 a year per line changed. Few code reviews look at a change with that cost in mind. And the signal pointed straight back at the change that caused it:

  1. A dated step. Requests jumped on the day the pipelines went live and stayed up.
  2. An exact location. The new writes sat under two checkpoint folders, each named after its pipeline.
  3. An owner. The pipeline's code and its infrastructure have an owner we could name.
  4. A check. The next week's request count says whether the change worked.

A daily check on the bucket's request count flags this step within days of go-live, while the fix is still a fresh change in the pipeline's configuration.

A software factory is a set of agents that write a change, test it and ship it, and today they optimise for passing tests. Give them this signal after each deploy, as plainly as a failing test, and "the pipeline costs four times more since your last change" becomes something they can act on and verify.

This is the cost feedback loop Cloud ex Machina runs. It reads the bill and the runtime data every day, ties a cost step to the change and the pipeline behind it, puts a dollar figure on it, and measures the next bills after the fix. With or without a factory, it delivers these findings as application-layer FinOps recommendations, each with its saving and the team that owns the code.

Three questions for any streaming job​

  • How often does its source change? If the answer is hours, continuous mode buys freshness nobody uses.
  • How many shuffle partitions does it use, and who picked that number? If nobody did, it's 200.
  • Is its checkpoint in a versioned bucket? Then every cleanup leaves a new object to pay for, until a lifecycle rule removes it.