Skip to content

📊 Monitoring the inference stack


Monitoring architecture

flowchart TD
    subgraph Sources["Metric sources"]
        A["vLLM /metrics"]
        B["nvidia-smi"] --> C["nvidia-dcgm-exporter"]
        D["OS"] --> E["node-exporter"]
    end
    Sources --> P["Prometheus\n(scrape + store)"]
    P --> G["Grafana\n(dashboards + alerts)"]

1. Full Docker Compose stack

docker-compose.monitoring.yml
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.retention.time=30d'
ports:
- "9090:9090"
restart: unless-stopped
grafana:
image: grafana/grafana-oss:latest
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
environment:
- GF_SECURITY_ADMIN_PASSWORD=changeme
- GF_USERS_ALLOW_SIGN_UP=false
ports:
- "3000:3000"
restart: unless-stopped
depends_on:
- prometheus
nvidia-dcgm-exporter:
image: nvcr.io/nvidia/k8s/dcgm-exporter:latest
container_name: dcgm-exporter
runtime: nvidia
environment:
- NVIDIA_VISIBLE_DEVICES=all
ports:
- "9400:9400"
restart: unless-stopped
cap_add:
- SYS_ADMIN
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- '--path.procfs=/host/proc'
- '--path.sysfs=/host/sys'
ports:
- "9100:9100"
restart: unless-stopped
volumes:
prometheus_data:
grafana_data:
Fenêtre de terminal
docker compose -f docker-compose.monitoring.yml up -d

2. Prometheus configuration

prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'vllm'
static_configs:
- targets: ['host.docker.internal:8000'] # adjust if vLLM on another machine
metrics_path: '/metrics'
- job_name: 'nvidia-gpu'
static_configs:
- targets: ['dcgm-exporter:9400']
- job_name: 'node'
static_configs:
- targets: ['node-exporter:9100']
# If Ollama: no native /metrics — scrape the metrics of the reverse proxy placed in front of
# Ollama (Caddy: admin endpoint :2019/metrics; Nginx: nginx-prometheus-exporter)
- job_name: 'ollama-proxy'
static_configs:
- targets: ['host.docker.internal:2019'] # reverse proxy /metrics endpoint

vLLM’s /metrics endpoint is not protected by --api-key: expose it only to the Prometheus network. Minimum version: vLLM ≥ 0.34.0 (see Sécurité de l'inférence locale).


3. vLLM metrics — essentials

vLLM exposes metrics on GET /metrics1. Priority metrics to watch:

Throughput and queues

MetricDescriptionAlert threshold
vllm:num_requests_runningRequests in progress> 80% of --max-num-seqs
vllm:num_requests_waitingQueued requests> 0 for > 30 s
rate(vllm:generation_tokens_total[1m])Generation throughput in tokens/s (derived from the counter; the V0 gauge avg_generation_throughput_toks_per_s is gone)1< threshold defined by use case
vllm:prompt_tokens_totalPrompt tokens processed (cumulative)— (trend)
vllm:generation_tokens_totalGenerated tokens (cumulative)— (trend)
vllm:request_success_totalCompleted requests—

The vLLM documentation names counters without a suffix (vllm:generation_tokens, vllm:num_preemptions); the Prometheus exposition adds _total to them, hence vllm:generation_tokens_total in the queries above1.

KV Cache

MetricDescriptionAlert threshold
vllm:kv_cache_usage_percFraction of KV Cache used (1 = 100%)> 0.90
vllm:num_preemptions_totalPreempted requests (KV Cache full)> 0 regularly

Latencies

MetricDescription
vllm:time_to_first_token_secondsTTFT distribution per request
vllm:inter_token_latency_secondsLatency between two generated tokens (histogram)1
vllm:request_time_per_output_token_secondsAverage time per output token, per request (replaces vllm:time_per_output_token_seconds)1
vllm:e2e_request_latency_secondsEnd-to-end latency

4. GPU metrics (DCGM Exporter)

NVIDIA DCGM Exporter exposes detailed GPU metrics5:

DCGM metricDescriptionAlert threshold
DCGM_FI_DEV_GPU_UTILGPU utilization (%)< 10% for > 5 min (waste)
DCGM_FI_DEV_MEM_COPY_UTILMemory bus utilization (%)> 90% sustained
DCGM_FI_DEV_FB_USEDVRAM used (MiB)> 95% of total VRAM (except GPUs served by vLLM ≥ 0.34, which deliberately occupies almost all VRAM: monitor vllm:kv_cache_usage_perc instead)
DCGM_FI_DEV_GPU_TEMPGPU temperature (°C)> 83°C (likely throttling)
DCGM_FI_DEV_POWER_USAGEPower draw (W)> declared TDP
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTALAggregate NVLink throughput (MB/s)— (trend on clusters)

DCGM_FI_DEV_FB_TOTAL is not exported by default: add DCGM_FI_DEV_FB_TOTAL, gauge, Framebuffer total (MiB) to the counters file (etc/default-counters.csv, mounted in the container) or use DCGM_FI_DEV_FB_FREE in the VRAMAlmostFull alert5.

Prometheus query — VRAM used per GPU

DCGM_FI_DEV_FB_USED{gpu=~".*"}

Import from Grafana.com

  1. Open Grafana → Dashboards → Import
  2. Enter dashboard ID:
DashboardGrafana IDUsage
NVIDIA DCGM Exporter12239GPU utilization, memory, temperature6
Node Exporter Full1860Host CPU, RAM, network, disk7

Essential panels to build manually

Panel “Throughput tokens/s”:

rate(vllm:generation_tokens_total[1m])

Panel “KV Cache %”:

vllm:kv_cache_usage_perc * 100

Panel “Queue”:

vllm:num_requests_waiting

Panel “TTFT P95” (95th percentile):

histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))

6. Prometheus alerts

# alerts.yml — reference in prometheus.yml under rule_files:
groups:
- name: inference-alerts
rules:
- alert: VLLMKVCacheHigh
expr: vllm:kv_cache_usage_perc > 0.90
for: 2m
labels:
severity: warning
annotations:
summary: "vLLM KV Cache > 90%"
description: "KV Cache is at {{ $value | humanizePercentage }}. Risk of preemption."
- alert: VLLMRequestQueueBacklog
expr: vllm:num_requests_waiting > 0
for: 1m
labels:
severity: warning
annotations:
summary: "vLLM queue not empty"
description: "{{ $value }} requests waiting for more than 1 minute."
- alert: GPUTemperatureHigh
expr: DCGM_FI_DEV_GPU_TEMP > 83
for: 5m
labels:
severity: critical
annotations:
summary: "GPU {{ $labels.gpu }} overheating"
description: "GPU temperature at {{ $value }}°C — likely throttling."
- alert: VRAMAlmostFull
# On GPUs served by vLLM ≥ 0.34, which deliberately reserves almost all VRAM, this alert would fire permanently: exclude those GPUs and monitor vllm:kv_cache_usage_perc instead (VLLMKVCacheHigh alert)
# DCGM_FI_DEV_FB_TOTAL must be added to the counters file (section 4);
# otherwise: DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 0.95
for: 2m
labels:
severity: critical
annotations:
summary: "GPU {{ $labels.gpu }} VRAM critical"
description: "{{ $value | humanizePercentage }} VRAM used."

To enable alerts in Prometheus:

# prometheus.yml — add
rule_files:
- "alerts.yml"
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093'] # if Alertmanager deployed

7. Monitoring Ollama (without native Prometheus)

If you use Ollama, a few immediate monitoring commands9:

Fenêtre de terminal
# Loaded models and VRAM used
ollama ps
# Real-time logs (include generation durations) — there is no `ollama logs` subcommand
# Linux (systemd)
journalctl -u ollama --no-pager --follow --pager-end
# macOS: cat ~/.ollama/logs/server.log — Docker: docker logs <container>
# Metrics in API response (durations in nanoseconds)
curl -s http://localhost:11434/api/generate \
-d '{"model":"llama3.2","prompt":"test","stream":false}' \
| python3 -c "
import sys, json
r = json.load(sys.stdin)
print(f'TTFT: {r[\"prompt_eval_duration\"]/1e9:.2f}s')
print(f'Generation: {r[\"eval_duration\"]/1e9:.2f}s')
print(f'Throughput: {r[\"eval_count\"]/(r[\"eval_duration\"]/1e9):.1f} tok/s')
"

8. Traces vs. metrics — OpenTelemetry for agent stacks

Prometheus and Grafana answer “Is the server healthy?”. They do not answer “Why did this request take 30 seconds?”.

The agent problem

In a RAG or multi-agent pipeline, one request crosses several services before producing an answer. If a user complains about 30s latency, Grafana may show the GPU at 70% utilization — not enough to diagnose the cause. You need the full request trace:

User request (30s total)
├── VectorDB Qdrant — 4s (semantic search)
├── Agent routing — 2s (tool selection)
├── SearXNG web search — 20s (network timeout ⚠️)
└── LLM generation — 4s (200 tokens @ 50 tok/s)

Without traces, engineers search logs in each service separately. With OpenTelemetry traces, the full request appears as one waterfall in a dedicated tool.

OpenTelemetry (OTEL) — native vLLM and LiteLLM integration

LiteLLM and vLLM support OpenTelemetry natively10:

Fenêtre de terminal
# vLLM — enable OTEL (export to local collector)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--otlp-traces-endpoint http://localhost:4317
# --collect-detailed-traces model for detailed traces (costly; all|model|worker)[^11]
# LiteLLM — litellm_config.yaml
litellm_settings:
callbacks: ["otel"]
# and in the LiteLLM proxy environment:
# OTEL_EXPORTER=otlp_grpc
# OTEL_ENDPOINT=http://localhost:4317 # OTLP gRPC

For a sovereign on-prem infrastructure, three options:

ToolTypeStrengthsDeployment
Langfuse (self-hosted, v4)Traces + LLM evalLLM-focused UI, cost per token, quality evaluationOfficial Docker Compose (local use and testing); Helm in production11
JaegerDistributed tracesCNCF standard, lightweight, Kubernetes-readyDocker jaegertracing/all-in-one
Arize PhoenixTraces + agent debugSpecialized for agents/RAG, free; Elastic License 2.0 (source-available, not OSI, forbids offering it as a managed service)pip install arize-phoenix1213

Minimal recommended stack for a custodian agent:

# docker-compose.yml — add to existing stack
# Langfuse (v4) is no longer a single container: it needs PostgreSQL, ClickHouse,
# Redis/Valkey and S3 storage (MinIO locally). Start from the official docker-compose.yml
# in the Langfuse repository (positioned as "local use and testing"); for
# production, the Helm chart. See note [^6].
# git clone https://github.com/langfuse/langfuse && cd langfuse && docker compose up -d
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
volumes:
- ./otel-config.yaml:/etc/otel-config.yaml
command: ["--config=/etc/otel-config.yaml"]
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP

Application state backup (DRP)

Monitoring detects failures — but without backups, recovery after an incident can take hours or lose data. Here is the state to protect in a local inference stack.

What to back up

ComponentCritical dataRecommended method
Qdrant (vector DB)Collections + index snapshotsPOST /collections/{name}/snapshots → local snapshot; copy to external storage
Milvus (vector DB)Collections, segments, metadataMilvus Backup CLI (milvus-backup create) to local S3 or NAS
SQLite (Memory Tree, histories).db fileDaily cp with rotation or hot sqlite3 .backup
vLLM / Ollama configconfig.yaml, startup scriptsVersioned in Git — already covered
Models and adaptersGGUF weights, fine-tuned LoRABase weights = immutable (re-download); fine-tuned adapters = back up to NAS

Suggested frequencies

Qdrant snapshot → daily (cron 02:00), 7-day retention
Milvus backup → daily (cron 03:00), 7-day retention
SQLite Memory Tree → hourly (if frequent writes), otherwise daily
LoRA adapters → after each training session

Qdrant snapshot example (curl)

Fenêtre de terminal
# Create a snapshot
curl -X POST http://localhost:6333/collections/my_collection/snapshots
# List available snapshots
curl http://localhost:6333/collections/my_collection/snapshots
# Copy to external storage (mounted NAS example)
cp /qdrant/snapshots/my_collection/*.snapshot /mnt/backup/qdrant/

Recovery procedure (indicative RTO)

  1. Restart the container vLLM or Ollama → reload weights from SSD (~15–120 s depending on model)
  2. Restore Qdrant from latest snapshot → PUT /collections/{name}/snapshots/recover
  3. Restore SQLite → copy backup .db over the corrupted file
  4. Verify health endpoint (/health or /v1/models)

See also


Sources and references

Footnotes

  1. vLLM Project, Metrics (/metrics endpoint, full list of exposed Prometheus metrics — vllm:kv_cache_usage_perc, vllm:num_preemptions, vllm:simple_kv_offload_* — metric deprecation policy), accessed 2026-10-09. https://docs.vllm.ai/en/stable/usage/metrics/ ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8

  2. Ollama, API Reference (/api/generate, /api/chat endpoints and eval_count / eval_duration / prompt_eval_duration fields; no /metrics endpoint), accessed 2026-10-09. https://docs.ollama.com/api ↩

  3. vLLM Project, Engine Arguments (--gpu-memory-utilization, --max-model-len, --max-num-seqs), accessed 2026-10-09. https://docs.vllm.ai/en/stable/configuration/engine_args/ ↩

  4. vLLM Project, Disaggregated Prefill and Decode (--kv-transfer-config, OffloadingConnector for KV Cache CPU offload), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/disagg_prefill/ ↩

  5. NVIDIA, DCGM Exporter (etc/default-counters.csv: GPU_UTIL, MEM_COPY_UTIL, FB_USED, FB_FREE, GPU_TEMP, POWER_USAGE, NVLINK_BANDWIDTH_TOTAL in MB/s; FB_TOTAL absent from default counters), accessed 2026-10-09. https://github.com/NVIDIA/dcgm-exporter ↩ ↩2

  6. Grafana Labs, NVIDIA DCGM Exporter Dashboard (ID 12239, GPU metrics visualization). https://grafana.com/grafana/dashboards/12239-nvidia-dcgm-exporter-dashboard/ ↩

  7. Grafana Labs, Node Exporter Full Dashboard (ID 1860, system metrics — CPU, memory, disk, network). https://grafana.com/grafana/dashboards/1860-node-exporter-full/ ↩

  8. vLLM Project, vllm-project/vllm repository, examples/observability/prometheus_grafana/ (README, docker-compose.yaml, prometheus.yaml, grafana.json) and examples/observability/dashboards/grafana/ (performance_statistics.json, query_statistics.json), main tree accessed 2026-10-10. https://github.com/vllm-project/vllm/tree/main/examples/observability/prometheus_grafana ↩

  9. Ollama, Troubleshooting (log locations: journalctl -u ollama, ~/.ollama/logs/server.log, docker logs), accessed 2026-10-09 · ollama/ollama repository, cmd/cmd.go (subcommand list, no logs). https://docs.ollama.com/troubleshooting · https://github.com/ollama/ollama ↩

  10. LiteLLM, OpenTelemetry Integration (litellm_settings.callbacks: ["otel"], OTEL_EXPORTER / OTEL_ENDPOINT variables), accessed 2026-10-09 · vLLM Project, CLI Reference — vllm serve (--otlp-traces-endpoint, --collect-detailed-traces all|model|worker), accessed 2026-10-09. https://docs.litellm.ai/docs/observability/opentelemetry_integration · https://docs.vllm.ai/en/stable/cli/serve/ ↩

  11. Langfuse, Self-Hosting (v4: PostgreSQL + ClickHouse + Redis/Valkey + S3 storage; Docker Compose for local use, Helm for production), accessed 2026-10-09. https://langfuse.com/self-hosting ↩

  12. Arize AI, Arize Phoenix — LLM Observability (agent traces, RAG debugging, OTEL-compatible), accessed 2026-10-09. https://arize.com/docs/phoenix ↩

  13. Arize AI, Arize-ai/phoenix repository, LICENSE file (Elastic License 2.0) and Release arize-phoenix-v20.0.0, 2026-08-11. https://github.com/Arize-ai/phoenix/blob/main/LICENSE · https://github.com/Arize-ai/phoenix/releases/tag/arize-phoenix-v20.0.0 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.