Skip to content

💰 TCO Comparison: On-Premise vs Cloud API


TCO calculation parameters

Before comparing, align the units. LLM usage is measured in millions of tokens processed per month — that is the cloud billing unit, and it is also the right denominator for calculating on-premise cost.

Cloud side: providers bill per token (input + output separately). Reference rates captured in October 2026, from budget model to frontier model:

API (prices captured 2026-10-09, USD excluding tax)Input rateOutput rateModel
OpenAI GPT-5.51$5.00/M tok$30.00/M tokProprietary, frontier
OpenAI GPT-5.6 Sol (promo until 2026-11-30)1$4.00/M tok$20.00/M tokProprietary, high-end
OpenAI GPT-6 Sol1$2.00/M tok$10.00/M tokProprietary, mid-range
OpenAI GPT-6 Luna1$0.10/M tok$0.50/M tokProprietary, budget
OpenAI GPT-4o (legacy)1$2.50/M tok$10.00/M tokProprietary, previous generation
Anthropic Claude Fable 5.2 / Mythos 5.2 (since 2026-11-09)2$8.00/M tok$40.00/M tokProprietary, frontier + (Mythos 5.2 on request, via Anthropic’s cyber and life-sciences verification programs)
Anthropic Claude Opus 5.52$4.00/M tok$20.00/M tokProprietary, high-end
Anthropic Claude Sonnet 5.52$2.00/M tok$10.00/M tokProprietary, mid-range
Anthropic Claude Haiku 5.5 (prompt ≤ 100k)2$0.10/M tok$0.50/M tokProprietary, budget
Mistral Large 3 (deprecated since 2026-10-31, replaced by Mistral Large 4)3$0.50/M tok$1.50/M tokProprietary (open weights)
Mistral Large 4 (API, price since the weights release on 2026-10-29)3$0.70/M tok ($0.07 cached)$2.10/M tokOpen weights (Mistral Ecosystem Open-Weight license, modified MIT)
Groq (openai/gpt-oss-120b)4$0.15/M tok$0.60/M tokOpen weights, cloud (Llama 3.3 70B: quote-based pricing since 2026)
Together AI (Llama 3.3 70B)5$1.04/M tok$1.04/M tokOpen weights, cloud
DeepSeek V4.1 Flash (DeepSeek API, deepseek-flash model, peak hours; captured 2026-10-10)6$0.30/M tok$1.20/M tokOpen weights, Chinese cloud — half price off-peak; data processed in China, outside the EU and the United States (see Sovereignty & Privacy)
Open-weights cloud API (70B–300B MoE)45~$0.15–1.40/M tok~$0.30–4.40/M tokTogether AI / Groq range, October 2026

Note: prices change frequently. Verify current rates before building a business case.

On-premise side: cost is fixed (hardware amortization) + variable (electricity, maintenance). No per-token billing.


Reference scenario for comparison

To make the comparison concrete, use a typical SME case:

  • Usage: 10 active users, ~50 requests/day/user
  • Average exchange size: ~1,000 input tokens + ~500 output tokens
  • Monthly volume: ~22,500 exchanges × 1,500 tokens = ~33.75 M tokens/month
  • Target model: Quantized 70B (Q4_K_M) — quality sufficient for most business cases

Blueprint A — Dev Lab (used 24 GB GPU or 64 GB unified memory)

Suitable usage: solo developer or team of 2–3 people, 8B–14B models.

ItemAmount
Hardware (used RTX 3090/4090 PC, or Mac Studio M5 Max 64 GB at €3,659 incl. VAT, or Framework Desktop 64 GB at €2,209 incl. VAT — French prices captured 2026-10-09)78€1,500 – €3,700 (4-year amortization)
Monthly amortization~€30 – €75/month
Electricity (150 W × 8 h/day × 30 days × €0.20/kWh, regulated tariff August 2026)9~€7/month
Total monthly cost~€40 – €85/month

Cloud equivalent (5 M input tokens + 2.5 M output tokens per month, October 2026 prices; Fable 5.2 captured 2026-11-20):

  • Haiku 5.5 or GPT-6 Luna ($0.10 / $0.50): 0.5 + 1.25 = $1.75/month (€2)
  • Sonnet 5.5 or GPT-6 Sol ($2 / $10): 10 + 25 = $35/month (~€32)
  • GPT-5.5 ($5 / $30): 25 + 75 = $100/month (~€92)
  • Claude Fable 5.2 ($8 / $40): 40 + 100 = $140/month (~€129)

Blueprint B — SME Appliance (Mac Studio / 128 GB APU)

Suitable usage: 10 to 50 users, 70B model, maximum confidentiality.

ItemAmount
128 GB appliance (Framework Desktop 128 GB at €3,889 incl. VAT, out of stock · Mac Studio M5 Max 128 GB, price on Apple’s configurator · DGX Spark 128 GB ≈ $6,950 ≈ €6,400 — captured 2026-10-09)7810€3,900 – €6,400, €4,500 retained (4-year amortization)
Monthly amortization~€95/month
Electricity (100 W × 12 h/day × 30 days × €0.20/kWh, regulated tariff August 2026)9~€7/month
Maintenance, backup, support~€50/month
Total monthly cost~€155/month

Cloud equivalent (22.5 M input tokens + 11.25 M output tokens per month, prices of 2026-10-09; DeepSeek captured 2026-10-10, Fable 5.2 and Mistral Large 4 on 2026-11-20):

API choice (prices of 2026-10-09, $1 ≈ €0.92)Cloud cost/monthBreak-even
Budget: Haiku 5.5, GPT-6 Luna, Mistral Large 4 ($0.10–0.70 / $0.50–2.10)123~€7 – 36/month (Haiku 5.5 or GPT-6 Luna: 2.25 + 5.6 = ~$8)❌ Never amortized on cost alone
Chinese budget: DeepSeek V4.1 Flash, DeepSeek API ($0.30 / $1.20 at peak hours, half off-peak)6~€9 – 19/month (peak hours: 6.75 + 13.5 = ~$20)❌ Never amortized on cost alone
Open-weights cloud (Together AI Llama 3.3 70B, $1.04/M)5~€32/month❌ Never amortized on cost alone
Mid-range: Sonnet 5.5, GPT-6 Sol ($2 / $10)12~€145/month (45 + 112.5 = ~$158)~51 months (beyond the amortization period)
High-end: Opus 5.5, GPT-5.6 Sol ($4 / $20)12~€290/month~19 months
Frontier: GPT-5.5 ($5 / $30)1~€414/month (112.5 + 337.5 = ~$450)~13 months
Frontier +: Claude Fable 5.2 ($8 / $40)2~€580/month (180 + 450 = ~$630)~9 months

Break-even = €4,500 of capital / (monthly cloud cost − €57/month of on-premise running cost excluding amortization), formula from the “Calculate your own TCO” section: 4,500 / (580 − 57) ≈ 8.6 months against Fable 5.2; 4,500 / (414 − 57) ≈ 12.6 months against GPT-5.5.


Blueprint C — Desktop Cluster (Exo / Thunderbolt)

Suitable usage: prototyping models > 100B, asynchronous batch processing.

ItemAmount
4× Mac mini M5 Pro 64 GB (base 24 GB at €1,999 incl. VAT, 64 GB option on Apple’s configurator) or 4× Mac Studio M5 Max 64 GB at €3,659 incl. VAT — French prices captured 2026-10-097~€9,000 – €14,600 (4-year amortization)
Thunderbolt hub + cables~€300
Monthly amortization~€195 – €310/month
Electricity (4 × 30 W × 16 h/day × 30 days × €0.20/kWh, regulated tariff August 2026)9~€12/month
Maintenance and administration~€80/month
Total monthly cost~€285 – €400/month

Cloud equivalent for models > 100B:

Open-weights models in this class (DeepSeek V4, GLM-5.3, Kimi K3) are available via OpenAI-compatible APIs from specialized hosts (Together AI, Groq) at rates often lower than proprietary models — but hosted outside your premises5:

Service (prices of 2026-10-09; Lambda GPU cloud of 2026-11-19; USD excluding tax)100B+ rateCost for 33.75 M tok/month (2/3 input)
Together AI (DeepSeek V4 Pro)5$1.32 / $3.96/M tok~$74 ≈ €68/month
Together AI (DeepSeek V4.1 Flash)5$0.30 / $1.20/M tok~$20 ≈ €18/month
On-demand GPU cloud (4× A100 80 GB at $3.19/h, per-GPU price of the 8-GPU instance, or 4× H100 instance at $4.69/h, 24/7)11~$9,310 – 13,690/month~€8,600 – €12,600/month (≈ €2,900 – €4,200 at 8 h/day)

Calculation: €9,300 to €14,900 of capital / (€8,600/month of GPU cloud − €92/month of cluster running cost excluding amortization) ≈ 1.1 to 1.8 months.


Blueprint D — Datacenter (HGX 8-GPU)

Suitable usage: high-concurrency production, 50+ simultaneous users, strict SLA.

ItemAmount
8-GPU NVLink node (HGX H200 or B300 node: on quote, ≈ $300–420k; an 8× RTX PRO 6000 server, $266k OEM list price, costs less but links its GPUs over PCIe, without NVLink)12~€300,000 – €450,000, €400,000 used (5-year amortization)
Infrastructure (network, cooling) + electricity (~10 kW × 8,760 h × €0.15–0.20/kWh ≈ €13,000 – €17,500/year)9~€30,000 – €40,000/year, €35,000 used
Monthly hardware amortization~€6,700/month
Infrastructure + ops~€2,900/month
Dedicated infrastructure engineer (0.5 FTE)~€4,000/month
Total monthly cost~€13,600/month

Cloud equivalent for SaaS production 50+ users:

ServiceEstimated costComment
Cloud API (500 M tok/month, 2/3 input, prices of 2026-10-09; Fable 5.2 of 2026-11-20)12~$120 (Haiku 5.5) to ~$6,700/month (GPT-5.5); ~$9,300 (Fable 5.2)No custom SLA guarantee; the HGX node only becomes cheaper beyond ~1 G tokens/month against GPT-5.5 (~0.8 G against Fable 5.2)
Dedicated GPU cloud (8× A100 80 GB on-demand, $3.19/GPU-h)11~$18,600/month (8× H100 at $4.59/GPU-h: ~$26,800)Strong SLA, but high cost
Reserved GPU cloud (H100, Lambda 1-Click Clusters, 2 weeks to 1 year, 16 GPUs min.)11~$32,000 – 36,000/month scaled down to 8 GPUs ($5.54 – 6.16/GPU-h, unchanged as of 2026-11-19)Commitment; above on-demand pricing at Lambda: request a quote

Calculation: €400,000 / (€17,100/month of 8× A100 on-demand − €6,900/month of ops and infrastructure) ≈ 39 months; €400,000 / (€24,700/month of 8× H100 on-demand − €6,900) ≈ 22 months. API threshold: €13,600 ≈ $14,800/month ÷ $13.3/M weighted (GPT-5.5) ≈ 1.1 G tokens/month.


Summary — When to choose what?

Monthly token volume Sovereignty constraint → Recommended blueprint
─────────────────────────────────────────────────────────────────────────────
< 5 M tokens/month Low → Cloud API (cost < on-prem)
< 5 M tokens/month Strong (GDPR, trade secret) → Blueprint A or B
5–50 M tokens/month Moderate → Blueprint B if the comparator is a frontier model (GPT-5.5, Opus 5.5, Fable 5.2); otherwise API
5–50 M tokens/month Strong → Blueprint B mandatory
> 50 M tokens/month Any → Blueprint B or D
100B+ models Any → Blueprint C (prototyping) or D (prod)
50+ simultaneous users Strong → Blueprint D

3-year TCO — visual recap

BlueprintCost/month3-year totalEquivalent cloud API 3 years (October 2026 prices, Lambda GPU cloud of November 2026, $1 ≈ €0.92)
A (dev lab, 5 M in + 2.5 M out/month)~€40 – €85~€1,400 – €3,100~€60 (Haiku 5.5) / ~€1,160 (Sonnet 5.5, GPT-6 Sol) / ~€3,300 (GPT-5.5) / ~€4,600 (Fable 5.2)
B (SME, 34 M tok/month)~€155~€5,580~€260 (Haiku 5.5) / ~€670 (DeepSeek V4.1 Flash) / ~€1,160 (Together AI Llama 3.3 70B) / ~€5,200 (Sonnet 5.5) / ~€10,400 (Opus 5.5) / ~€14,900 (GPT-5.5) / ~€20,900 (Fable 5.2)
C (cluster, 34 M tok/month, 100B+ models)~€285 – €400~€10,300 – €14,400~€670 (DeepSeek V4.1 Flash) / ~€2,460 (DeepSeek V4 Pro, Together AI) / ~€309,000 (4× A100 on-demand 24/7)
D (datacenter, 500 M tok/month)~€13,600~€489,600~€77,000 (Sonnet 5.5) / ~€221,000 (GPT-5.5) / ~€309,000 (Fable 5.2) / ~€617,000 (8× A100 on-demand) / ~€888,000 (8× H100 on-demand) / ~€1.07M (8× H100 at Lambda 1-Click reserved rate, $5.54/GPU-h)

Calculation: monthly cloud cost from sections A to D × 36 months (DeepSeek captured 2026-10-10, Fable 5.2 on 2026-11-20); GPU cloud: 8× A100 on-demand ≈ $18,630/month at $3.19/GPU-h, 8× H100 on-demand ≈ $26,810/month at $4.59/GPU-h and 8× H100 at the 1-Click reserved rate ≈ $32,350/month at $5.54/GPU-h (Lambda, captured 2026-11-19)11.


Software FinOps: reduce cost per request before hardware

Before investing in more GPUs, two software optimizations can divide the real per-token cost by a significant factor:

1. RAG pre-filtering: by limiting the context sent to the LLM to the K best results (Top-3 instead of Top-20), input tokens are reduced by a factor of 5 to 10 without perceptible quality degradation. On a cloud API billed per input token, the savings are direct. On a local model, that frees VRAM and compute time. See RAG & Agents — FinOps section.

2. CPU/GPU routing: offloading embeddings and voice transcription (Whisper) to CPU frees all GPU VRAM for generation. On a 2× L40S server, this routing frees a few GB of VRAM for the KV Cache, hence a few more simultaneous users, without changing hardware: the real gain depends on the load and must be measured.

These two levers apply to all blueprints, but their impact is strongest on Blueprints B and D where multi-user concurrency is the sizing factor.


Calculate your own TCO

To build your business case, collect this data:

  1. Token volume/month: estimate from number of users × requests/day × tokens per exchange
  2. Reference cloud rate: identify the API matching your required quality level
  3. Hardware amortization: hardware price / amortization period (36–60 months)
  4. Electricity cost: power in kW × hours/day × 30 × local kWh rate
  5. Ops cost: administrator time × daily rate
  6. Break-even point: (Hardware cost) / (Monthly cloud cost - Monthly on-prem cost)

Choosing hardware by phase

The TCO comparison above focuses mainly on inference (frozen model, text generation). The required hardware profile changes significantly depending on the model lifecycle phase.

PhaseMemory needSuitable hardware profileExample
Inference (frozen model, generation)Weights + KV cacheFast GPU with sufficient VRAMRTX 5090 (32 GB), L40S (48 GB), APU 128 GB
LoRA fine-tuning (adapters only)Weights + gradients + optimizer states (~2–3× inference)High-capacity unified memory or multi-GPUMac Studio M5 Ultra 256–512 GB, AMD Gorgon Halo 192 GB, DGX Spark 128 GB
Full fine-tuning (complete SFT)Very high — often 2–4× raw weights in FP16Multi-GPU server or datacenter2–4× A100 80 GB, or DGX Station
Full training (pre-training)Several hundred GB to several TBDatacenter clusters — beyond SMB on-prem scopeH100, HGX/DGX systems

The “tokens/s per k€” KPI for comparing inference options

To arbitrate between two inference hardware options, the ratio tokens per second per thousand euros invested (tokens/s/k€) is more meaningful than raw speed alone.

Example reading:

  • RTX 5090 (32 GB, MSRP $1,999, but ≥ $5,000 ≈ €4,600 in stores in September 2026)14: at ~60 tok/s on an 8B model → ~13 tok/s/k€ at the observed price, ~30 at MSRP — the ratio now depends as much on the shortage as on the silicon
  • Mac Studio M5 Max 128 GB (~€5,000, 614 GB/s): if it delivers ~12–15 tok/s on a 70B Q4 → ~2.5–3 tok/s/k€7

These two figures are consistent: the Mac Studio serves much larger models than the RTX 5090 (32 GB), so direct comparison only makes sense for the same model and same quantization.


See also


📚 Sources and References

Footnotes

  1. OpenAI, API pricing (GPT-5.5 $5 / $30, GPT-5.6 Sol $4 / $20 on promotion until 2026-11-30, GPT-6 Sol $2 / $10, GPT-6 Luna $0.10 / $0.50, GPT-4o legacy $2.50 / $10 per million tokens), captured 2026-10-09; end date of the GPT-5.6 Sol promotion re-checked on 2026-11-20. https://developers.openai.com/api/docs/pricing ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10

  2. Anthropic, Pricing (Claude Opus 5.5 $4 / $20, Sonnet 5.5 $2 / $10, Haiku 5.5 $0.10 / $0.50 per million tokens, captured 2026-10-09; Claude Fable 5.2 $8 / $40, captured 2026-11-20; Claude 3.5 Sonnet removed from the page); Anthropic, Claude Fable 5.2 and Mythos 5.2 ($8 / $40 per million tokens, versus $10 / $50 for Fable 5.1 and Mythos 5.1; Mythos 5.2 available via the cyber and life-sciences verification programs), 2026-11-09. https://platform.claude.com/docs/en/about-claude/pricing · https://www.anthropic.com/claude-fable-and-mythos-5-2 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9

  3. Mistral AI, Pricing (Mistral Large 3: $0.50 / $1.50 per million tokens), captured 2026-10-09; Mistral AI, Mistral Large 3 (model deprecated on 2026-10-31, replaced by Mistral Large 4), accessed 2026-11-20. https://mistral.ai/pricing · https://docs.mistral.ai/models/mistral-large-3-25-12 · Mistral AI, announcement of the Mistral Large 4 weights release ($0.70 input, $0.07 cached input, $2.10 output per million tokens, versus $1.36 / $4.18 during the 2026-10-06 preview), 2026-10-29. https://mistral.ai/news/mistral-large-4-oss/ · https://docs.mistral.ai/models/mistral-large-4-0 ↩ ↩2 ↩3

  4. Groq, GroqCloud — Models (openai/gpt-oss-120b $0.15 / $0.60 per million tokens; Llama 3.3 70B and Llama 3.1 8B moved to “Enterprise / Contact sales”, no public price), captured 2026-10-09. https://console.groq.com/docs/models ↩ ↩2

  5. Together AI, Pricing (Llama 3.3 70B $1.04 / $1.04; DeepSeek V4 Pro $1.32 / $3.96, DeepSeek V4.1 Flash $0.30 / $1.20, GLM-5.3 $1.40 / $4.40 and Kimi K3 $3 / $15 via OpenAI-compatible API), captured 2026-10-09. https://www.together.ai/pricing ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  6. DeepSeek, Models & Pricing (deepseek-flash, served by DeepSeek-V4.1-Flash: $0.30 / $1.20 per million tokens at peak hours — 01:00–04:00 and 06:00–10:00 UTC, Monday to Friday excluding Chinese public holidays — and half price the rest of the time; deepseek-v4-pro $1.32 / $3.96), captured 2026-10-10. https://api-docs.deepseek.com/quick_start/pricing · DeepSeek, Privacy Policy (“we directly collect, process and store your Personal Data in People’s Republic of China”), 2026-02-10. https://cdn.deepseek.com/policies/en-US/deepseek-privacy-policy.html ↩ ↩2

  7. Apple, Mac Studio and Mac mini — Apple Store France (Mac Studio M5 Max 64 GB €3,659 incl. VAT; Mac mini M5 Pro 24 GB €1,999 incl. VAT; 128 GB and 64 GB options on the configurator only), captured 2026-10-09. https://www.apple.com/fr/shop/buy-mac/mac-studio · https://www.apple.com/fr/shop/buy-mac/mac-mini ↩ ↩2 ↩3 ↩4

  8. Framework, Framework Desktop — AMD Ryzen AI Max+ 395 (64 GB €2,209 incl. VAT, 128 GB €3,889 incl. VAT, out of stock), captured 2026-10-09. https://frame.work/fr/fr/products/desktop-diy-amd-aimax300 ↩ ↩2

  9. EDF, Tarif Bleu — offre d’électricité au tarif réglementé (Base option: €0.2001 incl. VAT/kWh since 2026-08-01), captured 2026-10-09. https://particulier.edf.fr/fr/accueil/electricite-gaz/offres-electricite/tarif-bleu.html ↩ ↩2 ↩3 ↩4

  10. ServeTheHome, NVIDIA DGX Spark 64GB Launched and Big 128GB GB10 Price Increases (DGX Spark 128 GB ≈ $6,950, launched at $3,999; 64 GB version at $4,999 via OEMs), 2026-10-03. https://www.servethehome.com/nvidia-dgx-spark-64gb-launched-and-big-128gb-gb10-price-increases/ · NVIDIA Blog, NVIDIA DGX Spark 64GB Gives Developers More Ways to Build and Scale Local AI (same GB10, from $4,999, models up to 100B parameters, OEMs Acer, ASUS, Dell, Gigabyte, HP, MSI from Friday 23 October; two units linked over QSFP = 128 GB, models up to 200B), 2 October 2026. https://blogs.nvidia.com/blog/local-ai-dgx-spark-64gb-sync/ ↩ ↩2 ↩3

  11. Lambda, GPU Cloud Pricing (on-demand, price per GPU-hour: H100 SXM $4.59 in an 8-GPU instance and $4.69 in a 4-GPU instance, HGX A100 80 GB $3.19 in an 8-GPU instance; H100 reserved 1-Click Clusters 2 weeks to 1 year: $5.54 – 6.16), captured 2026-11-19. https://lambda.com/pricing ↩ ↩2 ↩3 ↩4 ↩5

  12. AMD, AAI 2026: AMD Delivers Full-Stack Compute for the Agentic AI Era (note 8: 8× RTX PRO 6000 server at an OEM list price of $265,928 as of 2026-07-16; MI350P server estimated at $327,238), 23 July 2026. https://ir.amd.com/news-events/press-releases/detail/1294/aai-2026-amd-delivers-full-stack-compute-for-the-agentic-ai-era ↩

  13. TrendForce, press release of 30 September 2026 (contract DRAM prices up 10–15% in Q4 2026, increases expected over the following quarters). https://www.trendforce.com/presscenter/news/20260930-13258.html ↩

  14. Tom’s Hardware, Nvidia’s RTX 5090 vanishes from online retail in the US (up to $9,500 from third-party sellers for an MSRP of $1,999), 2026-09-14. https://www.tomshardware.com/pc-components/gpus/nvidias-rtx-5090-vanishes-from-online-retail-in-the-us-third-party-sellers-now-demand-as-much-as-usd9-500-for-nvidias-fastest-gpu ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.