💰 TCO Comparison: On-Premise vs Cloud API
TCO calculation parameters
Before comparing, align the units. LLM usage is measured in millions of tokens processed per month — that is the cloud billing unit, and it is also the right denominator for calculating on-premise cost.
Cloud side: providers bill per token (input + output separately). Reference rates captured in October 2026, from budget model to frontier model:
| API (prices captured 2026-10-09, USD excluding tax) | Input rate | Output rate | Model |
|---|---|---|---|
| OpenAI GPT-5.51 | $5.00/M tok | $30.00/M tok | Proprietary, frontier |
| OpenAI GPT-5.6 Sol (promo until 2026-11-30)1 | $4.00/M tok | $20.00/M tok | Proprietary, high-end |
| OpenAI GPT-6 Sol1 | $2.00/M tok | $10.00/M tok | Proprietary, mid-range |
| OpenAI GPT-6 Luna1 | $0.10/M tok | $0.50/M tok | Proprietary, budget |
| OpenAI GPT-4o (legacy)1 | $2.50/M tok | $10.00/M tok | Proprietary, previous generation |
| Anthropic Claude Fable 5.2 / Mythos 5.2 (since 2026-11-09)2 | $8.00/M tok | $40.00/M tok | Proprietary, frontier + (Mythos 5.2 on request, via Anthropic’s cyber and life-sciences verification programs) |
| Anthropic Claude Opus 5.52 | $4.00/M tok | $20.00/M tok | Proprietary, high-end |
| Anthropic Claude Sonnet 5.52 | $2.00/M tok | $10.00/M tok | Proprietary, mid-range |
| Anthropic Claude Haiku 5.5 (prompt ≤ 100k)2 | $0.10/M tok | $0.50/M tok | Proprietary, budget |
| Mistral Large 3 (deprecated since 2026-10-31, replaced by Mistral Large 4)3 | $0.50/M tok | $1.50/M tok | Proprietary (open weights) |
| Mistral Large 4 (API, price since the weights release on 2026-10-29)3 | $0.70/M tok ($0.07 cached) | $2.10/M tok | Open weights (Mistral Ecosystem Open-Weight license, modified MIT) |
| Groq (openai/gpt-oss-120b)4 | $0.15/M tok | $0.60/M tok | Open weights, cloud (Llama 3.3 70B: quote-based pricing since 2026) |
| Together AI (Llama 3.3 70B)5 | $1.04/M tok | $1.04/M tok | Open weights, cloud |
DeepSeek V4.1 Flash (DeepSeek API, deepseek-flash model, peak hours; captured 2026-10-10)6 | $0.30/M tok | $1.20/M tok | Open weights, Chinese cloud — half price off-peak; data processed in China, outside the EU and the United States (see Sovereignty & Privacy) |
| Open-weights cloud API (70B–300B MoE)45 | ~$0.15–1.40/M tok | ~$0.30–4.40/M tok | Together AI / Groq range, October 2026 |
Note: prices change frequently. Verify current rates before building a business case.
On-premise side: cost is fixed (hardware amortization) + variable (electricity, maintenance). No per-token billing.
Reference scenario for comparison
To make the comparison concrete, use a typical SME case:
- Usage: 10 active users, ~50 requests/day/user
- Average exchange size: ~1,000 input tokens + ~500 output tokens
- Monthly volume: ~22,500 exchanges × 1,500 tokens = ~33.75 M tokens/month
- Target model: Quantized 70B (Q4_K_M) — quality sufficient for most business cases
Blueprint A — Dev Lab (used 24 GB GPU or 64 GB unified memory)
Suitable usage: solo developer or team of 2–3 people, 8B–14B models.
| Item | Amount |
|---|---|
| Hardware (used RTX 3090/4090 PC, or Mac Studio M5 Max 64 GB at €3,659 incl. VAT, or Framework Desktop 64 GB at €2,209 incl. VAT — French prices captured 2026-10-09)78 | €1,500 – €3,700 (4-year amortization) |
| Monthly amortization | ~€30 – €75/month |
| Electricity (150 W × 8 h/day × 30 days × €0.20/kWh, regulated tariff August 2026)9 | ~€7/month |
| Total monthly cost | ~€40 – €85/month |
Cloud equivalent (5 M input tokens + 2.5 M output tokens per month, October 2026 prices; Fable 5.2 captured 2026-11-20):
- Haiku 5.5 or GPT-6 Luna ($0.10 / $0.50): 0.5 + 1.25 =
$1.75/month (€2) - Sonnet 5.5 or GPT-6 Sol ($2 / $10): 10 + 25 = $35/month (~€32)
- GPT-5.5 ($5 / $30): 25 + 75 = $100/month (~€92)
- Claude Fable 5.2 ($8 / $40): 40 + 100 = $140/month (~€129)
Blueprint B — SME Appliance (Mac Studio / 128 GB APU)
Suitable usage: 10 to 50 users, 70B model, maximum confidentiality.
| Item | Amount |
|---|---|
| 128 GB appliance (Framework Desktop 128 GB at €3,889 incl. VAT, out of stock · Mac Studio M5 Max 128 GB, price on Apple’s configurator · DGX Spark 128 GB ≈ $6,950 ≈ €6,400 — captured 2026-10-09)7810 | €3,900 – €6,400, €4,500 retained (4-year amortization) |
| Monthly amortization | ~€95/month |
| Electricity (100 W × 12 h/day × 30 days × €0.20/kWh, regulated tariff August 2026)9 | ~€7/month |
| Maintenance, backup, support | ~€50/month |
| Total monthly cost | ~€155/month |
Cloud equivalent (22.5 M input tokens + 11.25 M output tokens per month, prices of 2026-10-09; DeepSeek captured 2026-10-10, Fable 5.2 and Mistral Large 4 on 2026-11-20):
| API choice (prices of 2026-10-09, $1 ≈ €0.92) | Cloud cost/month | Break-even |
|---|---|---|
| Budget: Haiku 5.5, GPT-6 Luna, Mistral Large 4 ($0.10–0.70 / $0.50–2.10)123 | ~€7 – 36/month (Haiku 5.5 or GPT-6 Luna: 2.25 + 5.6 = ~$8) | ❌ Never amortized on cost alone |
| Chinese budget: DeepSeek V4.1 Flash, DeepSeek API ($0.30 / $1.20 at peak hours, half off-peak)6 | ~€9 – 19/month (peak hours: 6.75 + 13.5 = ~$20) | ❌ Never amortized on cost alone |
| Open-weights cloud (Together AI Llama 3.3 70B, $1.04/M)5 | ~€32/month | ❌ Never amortized on cost alone |
| Mid-range: Sonnet 5.5, GPT-6 Sol ($2 / $10)12 | ~€145/month (45 + 112.5 = ~$158) | ~51 months (beyond the amortization period) |
| High-end: Opus 5.5, GPT-5.6 Sol ($4 / $20)12 | ~€290/month | ~19 months |
| Frontier: GPT-5.5 ($5 / $30)1 | ~€414/month (112.5 + 337.5 = ~$450) | ~13 months |
| Frontier +: Claude Fable 5.2 ($8 / $40)2 | ~€580/month (180 + 450 = ~$630) | ~9 months |
Break-even = €4,500 of capital / (monthly cloud cost − €57/month of on-premise running cost excluding amortization), formula from the “Calculate your own TCO” section: 4,500 / (580 − 57) ≈ 8.6 months against Fable 5.2; 4,500 / (414 − 57) ≈ 12.6 months against GPT-5.5.
Blueprint C — Desktop Cluster (Exo / Thunderbolt)
Suitable usage: prototyping models > 100B, asynchronous batch processing.
| Item | Amount |
|---|---|
| 4× Mac mini M5 Pro 64 GB (base 24 GB at €1,999 incl. VAT, 64 GB option on Apple’s configurator) or 4× Mac Studio M5 Max 64 GB at €3,659 incl. VAT — French prices captured 2026-10-097 | ~€9,000 – €14,600 (4-year amortization) |
| Thunderbolt hub + cables | ~€300 |
| Monthly amortization | ~€195 – €310/month |
| Electricity (4 × 30 W × 16 h/day × 30 days × €0.20/kWh, regulated tariff August 2026)9 | ~€12/month |
| Maintenance and administration | ~€80/month |
| Total monthly cost | ~€285 – €400/month |
Cloud equivalent for models > 100B:
Open-weights models in this class (DeepSeek V4, GLM-5.3, Kimi K3) are available via OpenAI-compatible APIs from specialized hosts (Together AI, Groq) at rates often lower than proprietary models — but hosted outside your premises5:
| Service (prices of 2026-10-09; Lambda GPU cloud of 2026-11-19; USD excluding tax) | 100B+ rate | Cost for 33.75 M tok/month (2/3 input) |
|---|---|---|
| Together AI (DeepSeek V4 Pro)5 | $1.32 / $3.96/M tok | ~$74 ≈ €68/month |
| Together AI (DeepSeek V4.1 Flash)5 | $0.30 / $1.20/M tok | ~$20 ≈ €18/month |
| On-demand GPU cloud (4× A100 80 GB at $3.19/h, per-GPU price of the 8-GPU instance, or 4× H100 instance at $4.69/h, 24/7)11 | ~$9,310 – 13,690/month | ~€8,600 – €12,600/month (≈ €2,900 – €4,200 at 8 h/day) |
Calculation: €9,300 to €14,900 of capital / (€8,600/month of GPU cloud − €92/month of cluster running cost excluding amortization) ≈ 1.1 to 1.8 months.
Blueprint D — Datacenter (HGX 8-GPU)
Suitable usage: high-concurrency production, 50+ simultaneous users, strict SLA.
| Item | Amount |
|---|---|
| 8-GPU NVLink node (HGX H200 or B300 node: on quote, ≈ $300–420k; an 8× RTX PRO 6000 server, $266k OEM list price, costs less but links its GPUs over PCIe, without NVLink)12 | ~€300,000 – €450,000, €400,000 used (5-year amortization) |
| Infrastructure (network, cooling) + electricity (~10 kW × 8,760 h × €0.15–0.20/kWh ≈ €13,000 – €17,500/year)9 | ~€30,000 – €40,000/year, €35,000 used |
| Monthly hardware amortization | ~€6,700/month |
| Infrastructure + ops | ~€2,900/month |
| Dedicated infrastructure engineer (0.5 FTE) | ~€4,000/month |
| Total monthly cost | ~€13,600/month |
Cloud equivalent for SaaS production 50+ users:
| Service | Estimated cost | Comment |
|---|---|---|
| Cloud API (500 M tok/month, 2/3 input, prices of 2026-10-09; Fable 5.2 of 2026-11-20)12 | ~$120 (Haiku 5.5) to ~$6,700/month (GPT-5.5); ~$9,300 (Fable 5.2) | No custom SLA guarantee; the HGX node only becomes cheaper beyond ~1 G tokens/month against GPT-5.5 (~0.8 G against Fable 5.2) |
| Dedicated GPU cloud (8× A100 80 GB on-demand, $3.19/GPU-h)11 | ~$18,600/month (8× H100 at $4.59/GPU-h: ~$26,800) | Strong SLA, but high cost |
| Reserved GPU cloud (H100, Lambda 1-Click Clusters, 2 weeks to 1 year, 16 GPUs min.)11 | ~$32,000 – 36,000/month scaled down to 8 GPUs ($5.54 – 6.16/GPU-h, unchanged as of 2026-11-19) | Commitment; above on-demand pricing at Lambda: request a quote |
Calculation: €400,000 / (€17,100/month of 8× A100 on-demand − €6,900/month of ops and infrastructure) ≈ 39 months; €400,000 / (€24,700/month of 8× H100 on-demand − €6,900) ≈ 22 months. API threshold: €13,600 ≈ $14,800/month ÷ $13.3/M weighted (GPT-5.5) ≈ 1.1 G tokens/month.
Summary — When to choose what?
Monthly token volume Sovereignty constraint → Recommended blueprint─────────────────────────────────────────────────────────────────────────────< 5 M tokens/month Low → Cloud API (cost < on-prem)< 5 M tokens/month Strong (GDPR, trade secret) → Blueprint A or B5–50 M tokens/month Moderate → Blueprint B if the comparator is a frontier model (GPT-5.5, Opus 5.5, Fable 5.2); otherwise API5–50 M tokens/month Strong → Blueprint B mandatory> 50 M tokens/month Any → Blueprint B or D100B+ models Any → Blueprint C (prototyping) or D (prod)50+ simultaneous users Strong → Blueprint D3-year TCO — visual recap
| Blueprint | Cost/month | 3-year total | Equivalent cloud API 3 years (October 2026 prices, Lambda GPU cloud of November 2026, $1 ≈ €0.92) |
|---|---|---|---|
| A (dev lab, 5 M in + 2.5 M out/month) | ~€40 – €85 | ~€1,400 – €3,100 | ~€60 (Haiku 5.5) / ~€1,160 (Sonnet 5.5, GPT-6 Sol) / ~€3,300 (GPT-5.5) / ~€4,600 (Fable 5.2) |
| B (SME, 34 M tok/month) | ~€155 | ~€5,580 | ~€260 (Haiku 5.5) / ~€670 (DeepSeek V4.1 Flash) / ~€1,160 (Together AI Llama 3.3 70B) / ~€5,200 (Sonnet 5.5) / ~€10,400 (Opus 5.5) / ~€14,900 (GPT-5.5) / ~€20,900 (Fable 5.2) |
| C (cluster, 34 M tok/month, 100B+ models) | ~€285 – €400 | ~€10,300 – €14,400 | ~€670 (DeepSeek V4.1 Flash) / ~€2,460 (DeepSeek V4 Pro, Together AI) / ~€309,000 (4× A100 on-demand 24/7) |
| D (datacenter, 500 M tok/month) | ~€13,600 | ~€489,600 | ~€77,000 (Sonnet 5.5) / ~€221,000 (GPT-5.5) / ~€309,000 (Fable 5.2) / ~€617,000 (8× A100 on-demand) / ~€888,000 (8× H100 on-demand) / ~€1.07M (8× H100 at Lambda 1-Click reserved rate, $5.54/GPU-h) |
Calculation: monthly cloud cost from sections A to D × 36 months (DeepSeek captured 2026-10-10, Fable 5.2 on 2026-11-20); GPU cloud: 8× A100 on-demand ≈ $18,630/month at $3.19/GPU-h, 8× H100 on-demand ≈ $26,810/month at $4.59/GPU-h and 8× H100 at the 1-Click reserved rate ≈ $32,350/month at $5.54/GPU-h (Lambda, captured 2026-11-19)11.
Software FinOps: reduce cost per request before hardware
Before investing in more GPUs, two software optimizations can divide the real per-token cost by a significant factor:
1. RAG pre-filtering: by limiting the context sent to the LLM to the K best results (Top-3 instead of Top-20), input tokens are reduced by a factor of 5 to 10 without perceptible quality degradation. On a cloud API billed per input token, the savings are direct. On a local model, that frees VRAM and compute time. See RAG & Agents — FinOps section.
2. CPU/GPU routing: offloading embeddings and voice transcription (Whisper) to CPU frees all GPU VRAM for generation. On a 2× L40S server, this routing frees a few GB of VRAM for the KV Cache, hence a few more simultaneous users, without changing hardware: the real gain depends on the load and must be measured.
These two levers apply to all blueprints, but their impact is strongest on Blueprints B and D where multi-user concurrency is the sizing factor.
Calculate your own TCO
To build your business case, collect this data:
- Token volume/month: estimate from number of users × requests/day × tokens per exchange
- Reference cloud rate: identify the API matching your required quality level
- Hardware amortization: hardware price / amortization period (36–60 months)
- Electricity cost: power in kW × hours/day × 30 × local kWh rate
- Ops cost: administrator time × daily rate
- Break-even point:
(Hardware cost) / (Monthly cloud cost - Monthly on-prem cost)
Choosing hardware by phase
The TCO comparison above focuses mainly on inference (frozen model, text generation). The required hardware profile changes significantly depending on the model lifecycle phase.
| Phase | Memory need | Suitable hardware profile | Example |
|---|---|---|---|
| Inference (frozen model, generation) | Weights + KV cache | Fast GPU with sufficient VRAM | RTX 5090 (32 GB), L40S (48 GB), APU 128 GB |
| LoRA fine-tuning (adapters only) | Weights + gradients + optimizer states (~2–3× inference) | High-capacity unified memory or multi-GPU | Mac Studio M5 Ultra 256–512 GB, AMD Gorgon Halo 192 GB, DGX Spark 128 GB |
| Full fine-tuning (complete SFT) | Very high — often 2–4× raw weights in FP16 | Multi-GPU server or datacenter | 2–4× A100 80 GB, or DGX Station |
| Full training (pre-training) | Several hundred GB to several TB | Datacenter clusters — beyond SMB on-prem scope | H100, HGX/DGX systems |
The “tokens/s per k€” KPI for comparing inference options
To arbitrate between two inference hardware options, the ratio tokens per second per thousand euros invested (tokens/s/k€) is more meaningful than raw speed alone.
Example reading:
- RTX 5090 (32 GB, MSRP $1,999, but ≥ $5,000 ≈ €4,600 in stores in September 2026)14: at ~60 tok/s on an 8B model → ~13 tok/s/k€ at the observed price, ~30 at MSRP — the ratio now depends as much on the shortage as on the silicon
- Mac Studio M5 Max 128 GB (~€5,000, 614 GB/s): if it delivers ~12–15 tok/s on a 70B Q4 → ~2.5–3 tok/s/k€7
These two figures are consistent: the Mac Studio serves much larger models than the RTX 5090 (32 GB), so direct comparison only makes sense for the same model and same quantization.
See also
- 🛠️ Scenario A — Dev Lab
- 🏢 Scenario B — SME Appliance
- 🖥️ Scenario C — Desktop Cluster
- 🏭 Scenario D — Datacenter
- 🔒 Sovereignty & Privacy
- 🧪 Evaluate a local model
📚 Sources and References
Footnotes
-
OpenAI, API pricing (GPT-5.5 $5 / $30, GPT-5.6 Sol $4 / $20 on promotion until 2026-11-30, GPT-6 Sol $2 / $10, GPT-6 Luna $0.10 / $0.50, GPT-4o legacy $2.50 / $10 per million tokens), captured 2026-10-09; end date of the GPT-5.6 Sol promotion re-checked on 2026-11-20. https://developers.openai.com/api/docs/pricing ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10
-
Anthropic, Pricing (Claude Opus 5.5 $4 / $20, Sonnet 5.5 $2 / $10, Haiku 5.5 $0.10 / $0.50 per million tokens, captured 2026-10-09; Claude Fable 5.2 $8 / $40, captured 2026-11-20; Claude 3.5 Sonnet removed from the page); Anthropic, Claude Fable 5.2 and Mythos 5.2 ($8 / $40 per million tokens, versus $10 / $50 for Fable 5.1 and Mythos 5.1; Mythos 5.2 available via the cyber and life-sciences verification programs), 2026-11-09. https://platform.claude.com/docs/en/about-claude/pricing · https://www.anthropic.com/claude-fable-and-mythos-5-2 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9
-
Mistral AI, Pricing (Mistral Large 3: $0.50 / $1.50 per million tokens), captured 2026-10-09; Mistral AI, Mistral Large 3 (model deprecated on 2026-10-31, replaced by Mistral Large 4), accessed 2026-11-20. https://mistral.ai/pricing · https://docs.mistral.ai/models/mistral-large-3-25-12 · Mistral AI, announcement of the Mistral Large 4 weights release ($0.70 input, $0.07 cached input, $2.10 output per million tokens, versus $1.36 / $4.18 during the 2026-10-06 preview), 2026-10-29. https://mistral.ai/news/mistral-large-4-oss/ · https://docs.mistral.ai/models/mistral-large-4-0 ↩ ↩2 ↩3
-
Groq, GroqCloud — Models (openai/gpt-oss-120b $0.15 / $0.60 per million tokens; Llama 3.3 70B and Llama 3.1 8B moved to “Enterprise / Contact sales”, no public price), captured 2026-10-09. https://console.groq.com/docs/models ↩ ↩2
-
Together AI, Pricing (Llama 3.3 70B $1.04 / $1.04; DeepSeek V4 Pro $1.32 / $3.96, DeepSeek V4.1 Flash $0.30 / $1.20, GLM-5.3 $1.40 / $4.40 and Kimi K3 $3 / $15 via OpenAI-compatible API), captured 2026-10-09. https://www.together.ai/pricing ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
DeepSeek, Models & Pricing (
deepseek-flash, served by DeepSeek-V4.1-Flash: $0.30 / $1.20 per million tokens at peak hours — 01:00–04:00 and 06:00–10:00 UTC, Monday to Friday excluding Chinese public holidays — and half price the rest of the time;deepseek-v4-pro$1.32 / $3.96), captured 2026-10-10. https://api-docs.deepseek.com/quick_start/pricing · DeepSeek, Privacy Policy (“we directly collect, process and store your Personal Data in People’s Republic of China”), 2026-02-10. https://cdn.deepseek.com/policies/en-US/deepseek-privacy-policy.html ↩ ↩2 -
Apple, Mac Studio and Mac mini — Apple Store France (Mac Studio M5 Max 64 GB €3,659 incl. VAT; Mac mini M5 Pro 24 GB €1,999 incl. VAT; 128 GB and 64 GB options on the configurator only), captured 2026-10-09. https://www.apple.com/fr/shop/buy-mac/mac-studio · https://www.apple.com/fr/shop/buy-mac/mac-mini ↩ ↩2 ↩3 ↩4
-
Framework, Framework Desktop — AMD Ryzen AI Max+ 395 (64 GB €2,209 incl. VAT, 128 GB €3,889 incl. VAT, out of stock), captured 2026-10-09. https://frame.work/fr/fr/products/desktop-diy-amd-aimax300 ↩ ↩2
-
EDF, Tarif Bleu — offre d’électricité au tarif réglementé (Base option: €0.2001 incl. VAT/kWh since 2026-08-01), captured 2026-10-09. https://particulier.edf.fr/fr/accueil/electricite-gaz/offres-electricite/tarif-bleu.html ↩ ↩2 ↩3 ↩4
-
ServeTheHome, NVIDIA DGX Spark 64GB Launched and Big 128GB GB10 Price Increases (DGX Spark 128 GB ≈ $6,950, launched at $3,999; 64 GB version at $4,999 via OEMs), 2026-10-03. https://www.servethehome.com/nvidia-dgx-spark-64gb-launched-and-big-128gb-gb10-price-increases/ · NVIDIA Blog, NVIDIA DGX Spark 64GB Gives Developers More Ways to Build and Scale Local AI (same GB10, from $4,999, models up to 100B parameters, OEMs Acer, ASUS, Dell, Gigabyte, HP, MSI from Friday 23 October; two units linked over QSFP = 128 GB, models up to 200B), 2 October 2026. https://blogs.nvidia.com/blog/local-ai-dgx-spark-64gb-sync/ ↩ ↩2 ↩3
-
Lambda, GPU Cloud Pricing (on-demand, price per GPU-hour: H100 SXM $4.59 in an 8-GPU instance and $4.69 in a 4-GPU instance, HGX A100 80 GB $3.19 in an 8-GPU instance; H100 reserved 1-Click Clusters 2 weeks to 1 year: $5.54 – 6.16), captured 2026-11-19. https://lambda.com/pricing ↩ ↩2 ↩3 ↩4 ↩5
-
AMD, AAI 2026: AMD Delivers Full-Stack Compute for the Agentic AI Era (note 8: 8× RTX PRO 6000 server at an OEM list price of $265,928 as of 2026-07-16; MI350P server estimated at $327,238), 23 July 2026. https://ir.amd.com/news-events/press-releases/detail/1294/aai-2026-amd-delivers-full-stack-compute-for-the-agentic-ai-era ↩
-
TrendForce, press release of 30 September 2026 (contract DRAM prices up 10–15% in Q4 2026, increases expected over the following quarters). https://www.trendforce.com/presscenter/news/20260930-13258.html ↩
-
Tom’s Hardware, Nvidia’s RTX 5090 vanishes from online retail in the US (up to $9,500 from third-party sellers for an MSRP of $1,999), 2026-09-14. https://www.tomshardware.com/pc-components/gpus/nvidias-rtx-5090-vanishes-from-online-retail-in-the-us-third-party-sellers-now-demand-as-much-as-usd9-500-for-nvidias-fastest-gpu ↩