A streaming pipeline turned a data lake bill into an API bill
· 6 min read
A data lake bucket held less than a terabyte and cost about $5 a day. On the day two new Databricks streaming pipelines went live, its cost rose to almost $19 a day, and S3 request costs made up 97% of it. The pipelines worked as written, the tables stayed fresh, and every dashboard stayed green. Two configuration defaults drove the extra cost, and the signal sat on the cloud bill, far from the pipeline code and its tests. A few lines of configuration fixed it, for about $5,000 a year. This post opens Feedback loops from the FinOps trenches, our series on runtime signals that point back at the code that caused them.

