The Bill That Will Not Stop Growing
A VP of Engineering at a Series C SaaS company opened her September 2025 cloud bill and saw inference costs had grown 340 percent in nine months. The product had grown 60 percent in users. The math did not work.
She was not unusual. IDC tracked global GenAI spend at $37 billion in 2024 and projected it to triple by 2028 (IDC, Worldwide AI Spending Guide, 2024). Most of that spend is now inference, not training. And most of it is uncontrolled.
The conversation she had with her CFO is happening in roughly half the engineering organizations I talk to. The bill grew faster than the user base because every product team turned on a model for their feature, picked the largest one because it gave the best demo, and shipped it to production with no cost ceiling and no fallback path.
This is not a technical problem. It is a FinOps problem with technical levers.
AI Velocity Blueprint
Measure and multiply engineering velocity using AI-powered diagnostics and sprint-aligned teams.
Coined Frame: The Five Levers
Five FinOps levers actually move the inference bill. Most teams pull one, maybe two. The ones who pull all five cut spend by 40 to 70 percent without touching product quality.
Lever 1 - Prompt caching. Anthropic shipped prompt caching for Claude in 2024 with reported cost reductions up to 90 percent for repeated long-context calls (Anthropic, "Prompt caching with Claude," 2024). If your application sends the same system prompt or document context across multiple user queries, caching cuts that token cost by an order of magnitude. Most teams have not turned it on.
Lever 2 - Tier routing. Not every request needs your strongest model. A well-designed router sends simple intent classification to a small fast model, sends complex reasoning to a frontier model, and only escalates the long-tail. BCG's 2024 study of enterprise AI cost optimization found tier routing saved 35 to 55 percent of model spend across their reviewed deployments (BCG, "AI at Scale," 2024).
Lever 3 - Batch shifting. Anthropic, OpenAI, and the major model APIs all offer batch endpoints at roughly 50 percent of synchronous pricing for non-interactive workloads. Most teams default to synchronous because they did not separate latency-sensitive from latency-tolerant traffic. Auditing your inference traffic for batch eligibility is usually a one-week engineering exercise with multi-month payback.
Lever 4 - Prompt economics. A 4,000-token prompt that could be 800 tokens is paying 5x for the same answer. Prompt compression, structured output schemas, and few-shot example pruning typically cut prompt size by 30 to 60 percent without quality loss. This is the lever most senior engineers know exists and most product teams have never had time to pull.
Lever 5 - Contract sizing. If you are spending more than $50K per month on a single model provider, you are leaving 20 to 40 percent on the table by paying list. Reserved capacity, committed spend agreements, and multi-model contracts are now standard at that scale. The procurement conversation pays for itself faster than any of the technical levers.
The Five Levers Sequenced
The order matters. Pulling them in the right sequence compounds.
Start with Lever 1, prompt caching, because it is mechanical and turns on in days. Move to Lever 5, contract sizing, because the savings unlock budget for the rest of the work. Then go to Lever 2, tier routing, because it requires real engineering investment but pays the most per dollar of effort. Lever 3, batch shifting, is a clean follow-up because it is mostly a queue and a scheduler. Lever 4, prompt economics, is iterative and benefits from being done last because the routing decisions in Lever 2 reshape what each prompt needs to do.
A team that sequences these correctly typically lands the first 25 percent of savings in 30 days and the rest over the following quarter.
The Forgotten Cost Categories
Three cost categories show up on the AI bill that most FinOps teams do not yet track separately. Each one needs a line item.
Embedding storage and retrieval. Vector databases at scale stop being a rounding error. Pinecone, Weaviate, pgvector at production volume crosses into mid-five-figure monthly territory fast. Index design choices made in week 2 of a project are now structural cost lines in year 2.
Evaluation and observability infrastructure. Running LLM evals, capturing traces, replaying production prompts against new model versions. This stack costs real money and is necessary, not optional. Treat it as a fixed cost line, the way you treat APM tooling.
Human review and HITL workflows. Wherever a human reviews AI output, that is a labor cost on the AI line. Most companies have not noticed yet because it is buried in headcount. Pulling it out into a transparent line item changes which workflows survive cost-benefit review.
What Does Not Work
Cost cutting that ignores quality has a half-life of about ninety days. If a router downgrades model tier and quality drops, product reverts the decision and trust in FinOps collapses.
The discipline that survives is FinOps with measured quality bars. Every cost lever sits next to a quality measurement. If quality drops past threshold, the lever flips back. The model tier router has fallback paths. The prompt compression has eval gates. The batch shifting has SLO tracking on the batch lane.
This is the difference between FinOps that engineering teams respect and FinOps that becomes an annual fight.
What This Looks Like on a P&L
A typical mid-market SaaS company spending $400K per month on inference today, after pulling the five levers in sequence, lands somewhere between $160K and $240K per month at the same quality bar. That is $2M to $3M per year that does not have to come from headcount cuts or feature deferrals.
The discount on contracts alone, for any company over $50K monthly spend, is usually enough to fund a dedicated FinOps engineer for a year. That hire then pays for itself two to four times over in the first year through the technical levers.
Evaluation Differnitator Framework
Why great CTOs don’t just build they evaluate. Use this framework to spot bottlenecks and benchmark performance.
Call to Action
What Logiciel Does Here
Logiciel runs cost-per-request engineering programs for CTOs whose AI features are now load-bearing on the P&L. We start with a baseline audit of where the spend actually goes, then sequence the five levers against your specific traffic profile and product constraints.
If you want a structured walkthrough before committing to a program, the AI Cost Per Request Framework covers the per-feature unit economics we use to scope these engagements. The CTO AI Evaluation Framework covers the quality bars that have to sit alongside any cost optimization work.
A 30-minute working session is usually enough to size the savings range on your current bill.
Frequently Asked Questions
What is the fastest cost lever to pull?
Prompt caching, if your application has repeated long-context calls. Most teams that turn it on see results within a billing cycle. The setup is hours, not weeks.
How do I know if I am paying too much for inference today?
Two heuristics. First, cost per active user per month: if it is over $3 for a SaaS product or over $0.50 for a consumer app, you are likely overpaying. Second, model tier ratio: if more than 30 percent of your calls go to your top-tier model, tier routing will likely cut 40 percent of cost.
Should I self-host models to cut inference cost?
Almost never below $500K monthly spend, and even above that the math is harder than the marketing suggests. GPU capacity, ops, model updates, and security all eat the headline savings. The teams winning at self-hosting usually have very narrow, very high-volume workloads.
How does FinOps work for agentic systems with unpredictable token usage?
Per-task budgets are the right primitive. Each agent run gets a budget ceiling at start, the system halts and escalates if it would exceed. Predictability comes from the budget, not from the model.
What is the right organizational owner for AI FinOps?
Joint ownership. Finance owns the budget envelope and contract negotiation. Engineering owns the technical levers and quality gates. A dedicated FinOps engineer, embedded in the platform team, is the right primary owner once spend crosses $200K per month. Sources: - IDC Worldwide AI Spending Guide, 2024 - Anthropic Prompt Caching announcement, 2024 - BCG "AI at Scale" enterprise cost study, 2024
