Rent tokens or run GPUs? Find your break-even.
Compare a pay-per-token API with self-hosting an open-weight model on cloud or owned GPUs. Set your volume, model size, GPU class, utilisation and the people needed to run it, and see which option is cheaper today, where the break-even sits, and when a hybrid makes more sense.
Compare API and GPU costs
Start from a preset, then set your volume, the API you would use and the GPUs you would run. Prices are editable.
- Pay-per-token API$1,800
- Self-hosted GPUs$17,000
- Pay-per-token API$0.880
- Self-hosted GPUs$8.66
- GPU rental$8,200
- Storage and networking$820
- Engineering and ops$8,300
- Self-hosting breaks even at around 25.4B tokens a month, about 13 times your current volume. Until then, stay on the API and revisit as usage grows.
- For confidential data, self-hosting keeps prompts inside your own environment for about $16,000 a month more than the API. Weigh that against residency rules and customer contracts.
- Engineering and operations are 48% of the self-hosted cost. This is the line teams most often underestimate: upgrades, monitoring, evaluations and on-call.
- Buying hardware would bring self-hosting to about $13,000 a month over 3 years, if you can keep utilisation high and have somewhere to run it.
- Serving optimisations (quantisation, continuous batching, speculative decoding) are assumed to lift throughput by about 40%. Without them you would need 4 GPUs instead of 4.
- An AI-native platform team using agents for runbooks, upgrades and incident triage could plausibly run this with a quarter less ops time, bringing self-hosting to about $15,000 a month. Senior engineers still own reliability.
- Compare quality on your own tasks, not just price. Open-weight models handle extraction, classification and grounded Q&A well; the gap to mid-tier api models is widest on complex reasoning.
Assumptions: 2 model copies of 2 GPUs each, running 730 hours a month; input tokens cost 1/8 of an output token in GPU time; 25% employment on-costs. Throughput and prices are indicative and vary with model, context length and serving stack.
Indicative only. We will send the inputs above with your message so a senior engineer can sanity-check them.
Four steps from token volume to a fair comparison
Most GPU vs API spreadsheets compare a GPU-hour price with a token price and stop there. This calculator adds the capacity, redundancy and staffing that decide the real answer.
Size the workload
VolumeMonthly tokens and the input to output split. Input tokens are cheap to process on a GPU, while generated tokens take most of the compute, so the split matters for both options.
Price the API
APIPick a generic tier and overwrite the prices with your own rates. Discounts from caching and batch processing, and any premium for enterprise data terms, are applied here.
Size the GPU fleet
GPUsModel size decides how many GPUs one copy of the model needs. Throughput per copy, target utilisation and redundancy then decide how many copies you run around the clock.
Add the people
OpsSelf-hosting needs engineers to deploy, upgrade, monitor and support the stack. The calculator adds that time at your salary level, because it often outweighs the hardware.
API vs self-hosted GPUs vs hybrid inference
Cost is only one axis. These are the differences that usually decide the architecture once the numbers are close.
| Pay-per-token API | Self-hosted GPUs | Hybrid | |
|---|---|---|---|
| Cost shape | Linear with usage, no idle cost | Mostly fixed, cheap at high utilisation | Fixed baseline plus usage for peaks |
| Best when | Low, spiky or uncertain volume | Steady, high, predictable volume | Steady baseline with bursts |
| Data control | Depends on vendor terms and region | Prompts stay in your environment | Sensitive traffic can stay on your GPUs |
| Model choice | Access to top frontier models | Open-weight models you can tune | Open-weight by default, frontier on demand |
| Operational load | Minimal | Serving stack, upgrades, on-call | Both, with routing logic between them |
| Biggest risk | Price or policy changes | Idle GPUs and underestimated ops | Complexity without enough volume |
Break-even usually arrives later than people expect.
GPU-hour prices look cheap next to an API invoice, but a production deployment rarely runs one GPU. You need enough copies of the model for peak traffic, at least one spare for failover, and engineers who understand the serving stack. At modest volumes those fixed costs dwarf the token bill, and API prices for comparable models have kept falling.
Self-hosting earns its place in three situations: sustained high volume where GPUs stay busy, data that cannot leave your environment, or a fine-tuned open-weight model that outperforms general APIs on your task. Hybrid setups are often the practical middle ground, running a steady baseline on your own GPUs and sending peaks to an API.
AI is changing the operations side as well. Teams that use AI agents for runbooks, dependency upgrades and incident triage can run inference infrastructure with less hands-on time, though senior engineers still need to own reliability, security and capacity planning.
Six checks before committing to self-hosted inference
Benchmark with your own prompts
Throughput varies with context length, batch size and serving software. Measure tokens per second on realistic traffic before sizing a fleet.
Test quality after quantisation
Quantised models are cheaper to serve but can lose accuracy on some tasks. Run your evaluation set on the exact model you plan to deploy.
Price the people honestly
Include on-call, upgrades, security patching and model evaluation. If nobody on the team has run a GPU serving stack, budget for learning or outside help.
Plan for GPU generations
Hardware generations turn over quickly. Rented capacity or shorter depreciation periods protect you from owning slow cards in three years.
Keep an API fallback
Even a well-run cluster has outages. Routing to an API when your GPUs are saturated or down protects users and avoids buying spare capacity.
Check what the API vendor offers first
Enterprise terms, regional hosting and dedicated capacity sometimes solve data concerns without taking on infrastructure.
Questions we often hear
Is it cheaper to self-host an LLM than use an API?
Only above a certain volume, and that point is often higher than expected. APIs have no idle cost, while self-hosting carries fixed GPU, redundancy and engineering costs from day one. Self-hosting becomes cheaper when GPUs stay busy with steady traffic, so utilisation matters as much as the hourly price.
How many tokens per month before self-hosting makes sense?
There is no single threshold. It depends on the API price you are replacing, the model size, the GPU class, utilisation and staffing. For teams replacing a low-cost mid-tier API it can take billions of tokens a month, while replacing an expensive frontier API with a capable open-weight model can break even much sooner.
What does it cost to run an LLM on H100-class GPUs?
Rental prices for H100-class GPUs vary widely by provider, region and commitment term, and have changed quickly as supply has grown. The calculator uses an indicative hourly default that you should replace with real quotes. Remember that a medium or large model usually needs several GPUs per copy, plus spare capacity.
Is self-hosting an LLM more private than using an API?
Self-hosting keeps prompts and outputs inside infrastructure you control, which simplifies data residency and retention questions. It is not automatically more secure, because you take on patching, access control and monitoring. Many API vendors also offer enterprise terms with no training on your data and regional processing, so compare both carefully.
What is a hybrid inference setup?
A hybrid setup runs a steady baseline of traffic on your own GPUs and sends overflow, rare hard requests or failover traffic to an API. It lets you keep GPUs highly utilised without buying capacity for peaks. It needs routing logic and monitoring, so it pays off once volume is meaningful.
Can AI tools reduce the cost of running self-hosted models?
They can reduce the operational effort. AI agents can draft runbooks, handle routine upgrades, summarise incidents and analyse capacity trends, which typically means fewer engineer-hours on routine work. Serving optimisations such as quantisation and continuous batching also raise throughput per GPU, but engineers still need to validate quality and own reliability.
Related reading and tools
Deciding between APIs and your own GPUs?
Send us your comparison and a little about your workload. A senior engineer will sanity-check the throughput, utilisation and staffing assumptions, and tell you honestly whether self-hosting, an API or a hybrid fits your situation.
Working with companies globally · Response within 24 hours