Keep the AI features. Lose the runaway inference bill.
Your AI feature worked in the pilot, then usage grew and the monthly API or GPU bill grew faster. We find where tokens and GPU hours are being wasted and apply the right levers, with an evaluation suite in place so cheaper never quietly means worse.
Why LLM inference costs grow faster than usage
Most AI features are priced per token, and tokens hide in places nobody budgets for. A long system prompt resent on every request, a retrieval step that stuffs twenty document chunks into context when three would do, a chat history that is never summarised, an agent that calls the model eight times to answer one question. Each is invisible in a demo and expensive at ten thousand users.
The second driver is model choice. Teams default to the most capable model during prototyping because it makes everything work, then never revisit the decision. In many production workloads a large share of requests are classification, extraction or short rewrites that a smaller, far cheaper model handles just as well.
Self-hosted models have their own version of the problem: GPUs sized for peak traffic that sit mostly idle, full-precision weights that need much more memory than the task requires, and serving setups that process one request at a time. The bill is fixed rather than per token, but the waste is just as real.
The main inference cost optimisation levers, compared
No single technique fixes an inference bill. These are the levers we reach for most often, and what each one asks of you in effort and risk.
| What it does | Best suited to | Watch out for | |
|---|---|---|---|
| Prompt and response caching | Reuses repeated prompt prefixes or whole answers instead of paying for them again | Long system prompts, FAQs, repeated document context | Stale answers if invalidation is careless |
| Model routing | Sends simple requests to small models and hard ones to large models | Mixed workloads with many easy requests | Needs evals to prove the small model is good enough |
| Context trimming | Cuts retrieved chunks, history and instructions to what the task needs | RAG, chat and agent workflows | Removing the context that made answers correct |
| Quantisation | Runs self-hosted models at lower numeric precision to use less GPU memory | Open-weight models on your own GPUs | Quality loss on some tasks, so test before and after |
| Batching | Groups requests so GPUs or provider batch APIs process them together | Offline jobs, bulk enrichment, busy self-hosted serving | Added latency, unsuitable for live chat as-is |
| API or self-hosted | Moves steady, high-volume workloads onto dedicated GPUs, or back to APIs | Predictable volume at scale | Operations burden and upfront commitment |
Provider pricing changes often. We model each lever against the current rates of your providers rather than rules of thumb from last year.
What we measure first
Cutting costs without measurement usually means cutting quality without noticing. These are the foundations we put in place before changing anything.
Cost per unit of value
Cost per conversation, per document processed or per active customer, not just the monthly total. This is the number that tells you whether a change worked.
A quality evaluation set
Several hundred real, anonymised requests with expected outcomes, scored automatically, so every cheaper configuration is compared against the current one before it ships.
Token accounting by feature and customer
Input, cached input and output tokens tagged at the source, so you can see which workflow or tenant is actually driving the bill.
Latency and error baselines
Some levers trade speed for cost. Knowing today's response times and failure rates stops a saving from turning into a support problem.
Agent and tool-call loops
Agentic workflows can call a model many times per task. We trace them to find retries, redundant steps and decisions that never needed a model at all.
An inference cost review in four stages
Most engagements run as a two to four week Advisory Sprint, with implementation by your team or ours. Our engineers use AI coding agents to build the instrumentation, eval harness and routing code quickly, and a senior engineer signs off every change to production behaviour.
Instrument
Week 1Add token and cost tracking per feature, pull provider invoices and GPU utilisation data, and build the evaluation set from real traffic.
Diagnose
Weeks 1 to 2Rank waste by monthly impact: oversized models, bloated context, cache misses, idle GPUs, chatty agents. You get a costed list, not a slide of generic tips.
Apply and test
Weeks 2 to 3Implement the highest-value levers behind feature flags, run each against the eval set and a slice of live traffic, and keep only what holds quality.
Hand over and monitor
Week 4Cost dashboards and budget alerts, documented decisions, and a simple routine for re-checking model choices when providers release new models or change prices.
Questions we often hear
How can I reduce my LLM API costs?
Start by measuring cost per feature, then tackle the biggest sources: route simple requests to smaller models, enable prompt caching for repeated context, trim retrieved chunks and chat history, and move non-urgent work to batch processing. Check every change against a quality evaluation set so the savings do not come at the expense of accuracy.
Is it cheaper to self-host an LLM than to use an API?
Sometimes. Self-hosting tends to pay off with high, steady volume and a model that runs efficiently on your GPUs, while spiky or low-volume workloads are usually cheaper on an API. Our GPU vs API inference calculator gives a first estimate of the break-even point, and staffing and operations costs belong in that comparison too.
Does quantisation reduce model quality?
It can. Moderate quantisation often has little visible effect on common business tasks, while aggressive quantisation is more likely to hurt reasoning, maths or multilingual output. The only reliable answer is to test the quantised model on your own evaluation set before switching.
What is model routing for LLMs?
Model routing sends each request to the cheapest model that can handle it well. A lightweight classifier or set of rules decides, for example, that a password reset question goes to a small model while a detailed contract question goes to a large one. Done with evaluations it lowers average cost per request; done without them it quietly lowers quality.
Can you cut inference costs without changing our AI provider?
Usually, yes. Caching, context trimming, output limits, routing between a provider's own model tiers and batch endpoints can all be applied within a single provider. We only recommend switching providers or self-hosting when the numbers clearly justify the extra work.
Related reading and tools
Tell us what your AI features cost today.
Share your providers, rough monthly spend and the features driving it. We will reply within 24 hours with where we would look first and whether a review is likely to pay for itself.
Working with companies globally · Response within 24 hours