📊 Monitoring the inference stack
Monitoring architecture
flowchart TD
subgraph Sources["Metric sources"]
A["vLLM /metrics"]
B["nvidia-smi"] --> C["nvidia-dcgm-exporter"]
D["OS"] --> E["node-exporter"]
end
Sources --> P["Prometheus\n(scrape + store)"]
P --> G["Grafana\n(dashboards + alerts)"]
1. Full Docker Compose stack
services:
prometheus: image: prom/prometheus:latest container_name: prometheus volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro - prometheus_data:/prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.retention.time=30d' ports: - "9090:9090" restart: unless-stopped
grafana: image: grafana/grafana-oss:latest container_name: grafana volumes: - grafana_data:/var/lib/grafana environment: - GF_SECURITY_ADMIN_PASSWORD=changeme - GF_USERS_ALLOW_SIGN_UP=false ports: - "3000:3000" restart: unless-stopped depends_on: - prometheus
nvidia-dcgm-exporter: image: nvcr.io/nvidia/k8s/dcgm-exporter:latest container_name: dcgm-exporter runtime: nvidia environment: - NVIDIA_VISIBLE_DEVICES=all ports: - "9400:9400" restart: unless-stopped cap_add: - SYS_ADMIN
node-exporter: image: prom/node-exporter:latest container_name: node-exporter volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro command: - '--path.procfs=/host/proc' - '--path.sysfs=/host/sys' ports: - "9100:9100" restart: unless-stopped
volumes: prometheus_data: grafana_data:docker compose -f docker-compose.monitoring.yml up -d2. Prometheus configuration
global: scrape_interval: 15s evaluation_interval: 15s
scrape_configs:
- job_name: 'vllm' static_configs: - targets: ['host.docker.internal:8000'] # adjust if vLLM on another machine metrics_path: '/metrics'
- job_name: 'nvidia-gpu' static_configs: - targets: ['dcgm-exporter:9400']
- job_name: 'node' static_configs: - targets: ['node-exporter:9100']
# If Ollama: no native /metrics — scrape the metrics of the reverse proxy placed in front of # Ollama (Caddy: admin endpoint :2019/metrics; Nginx: nginx-prometheus-exporter) - job_name: 'ollama-proxy' static_configs: - targets: ['host.docker.internal:2019'] # reverse proxy /metrics endpointvLLM’s /metrics endpoint is not protected by --api-key: expose it only to the Prometheus network. Minimum version: vLLM ≥ 0.34.0 (see Sécurité de l'inférence locale).
3. vLLM metrics — essentials
vLLM exposes metrics on GET /metrics1. Priority metrics to watch:
Throughput and queues
| Metric | Description | Alert threshold |
|---|---|---|
vllm:num_requests_running | Requests in progress | > 80% of --max-num-seqs |
vllm:num_requests_waiting | Queued requests | > 0 for > 30 s |
rate(vllm:generation_tokens_total[1m]) | Generation throughput in tokens/s (derived from the counter; the V0 gauge avg_generation_throughput_toks_per_s is gone)1 | < threshold defined by use case |
vllm:prompt_tokens_total | Prompt tokens processed (cumulative) | — (trend) |
vllm:generation_tokens_total | Generated tokens (cumulative) | — (trend) |
vllm:request_success_total | Completed requests | — |
The vLLM documentation names counters without a suffix (vllm:generation_tokens, vllm:num_preemptions); the Prometheus exposition adds _total to them, hence vllm:generation_tokens_total in the queries above1.
KV Cache
| Metric | Description | Alert threshold |
|---|---|---|
vllm:kv_cache_usage_perc | Fraction of KV Cache used (1 = 100%) | > 0.90 |
vllm:num_preemptions_total | Preempted requests (KV Cache full) | > 0 regularly |
Latencies
| Metric | Description |
|---|---|
vllm:time_to_first_token_seconds | TTFT distribution per request |
vllm:inter_token_latency_seconds | Latency between two generated tokens (histogram)1 |
vllm:request_time_per_output_token_seconds | Average time per output token, per request (replaces vllm:time_per_output_token_seconds)1 |
vllm:e2e_request_latency_seconds | End-to-end latency |
4. GPU metrics (DCGM Exporter)
NVIDIA DCGM Exporter exposes detailed GPU metrics5:
| DCGM metric | Description | Alert threshold |
|---|---|---|
DCGM_FI_DEV_GPU_UTIL | GPU utilization (%) | < 10% for > 5 min (waste) |
DCGM_FI_DEV_MEM_COPY_UTIL | Memory bus utilization (%) | > 90% sustained |
DCGM_FI_DEV_FB_USED | VRAM used (MiB) | > 95% of total VRAM (except GPUs served by vLLM ≥ 0.34, which deliberately occupies almost all VRAM: monitor vllm:kv_cache_usage_perc instead) |
DCGM_FI_DEV_GPU_TEMP | GPU temperature (°C) | > 83°C (likely throttling) |
DCGM_FI_DEV_POWER_USAGE | Power draw (W) | > declared TDP |
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL | Aggregate NVLink throughput (MB/s) | — (trend on clusters) |
DCGM_FI_DEV_FB_TOTAL is not exported by default: add DCGM_FI_DEV_FB_TOTAL, gauge, Framebuffer total (MiB) to the counters file (etc/default-counters.csv, mounted in the container) or use DCGM_FI_DEV_FB_FREE in the VRAMAlmostFull alert5.
Prometheus query — VRAM used per GPU
DCGM_FI_DEV_FB_USED{gpu=~".*"}5. Recommended Grafana dashboards
Import from Grafana.com
- Open Grafana → Dashboards → Import
- Enter dashboard ID:
| Dashboard | Grafana ID | Usage |
|---|---|---|
| NVIDIA DCGM Exporter | 12239 | GPU utilization, memory, temperature6 |
| Node Exporter Full | 1860 | Host CPU, RAM, network, disk7 |
Essential panels to build manually
Panel “Throughput tokens/s”:
rate(vllm:generation_tokens_total[1m])Panel “KV Cache %”:
vllm:kv_cache_usage_perc * 100Panel “Queue”:
vllm:num_requests_waitingPanel “TTFT P95” (95th percentile):
histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))6. Prometheus alerts
# alerts.yml — reference in prometheus.yml under rule_files:groups: - name: inference-alerts rules:
- alert: VLLMKVCacheHigh expr: vllm:kv_cache_usage_perc > 0.90 for: 2m labels: severity: warning annotations: summary: "vLLM KV Cache > 90%" description: "KV Cache is at {{ $value | humanizePercentage }}. Risk of preemption."
- alert: VLLMRequestQueueBacklog expr: vllm:num_requests_waiting > 0 for: 1m labels: severity: warning annotations: summary: "vLLM queue not empty" description: "{{ $value }} requests waiting for more than 1 minute."
- alert: GPUTemperatureHigh expr: DCGM_FI_DEV_GPU_TEMP > 83 for: 5m labels: severity: critical annotations: summary: "GPU {{ $labels.gpu }} overheating" description: "GPU temperature at {{ $value }}°C — likely throttling."
- alert: VRAMAlmostFull # On GPUs served by vLLM ≥ 0.34, which deliberately reserves almost all VRAM, this alert would fire permanently: exclude those GPUs and monitor vllm:kv_cache_usage_perc instead (VLLMKVCacheHigh alert) # DCGM_FI_DEV_FB_TOTAL must be added to the counters file (section 4); # otherwise: DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 0.95 for: 2m labels: severity: critical annotations: summary: "GPU {{ $labels.gpu }} VRAM critical" description: "{{ $value | humanizePercentage }} VRAM used."To enable alerts in Prometheus:
# prometheus.yml — addrule_files: - "alerts.yml"
alerting: alertmanagers: - static_configs: - targets: ['alertmanager:9093'] # if Alertmanager deployed7. Monitoring Ollama (without native Prometheus)
If you use Ollama, a few immediate monitoring commands9:
# Loaded models and VRAM usedollama ps
# Real-time logs (include generation durations) — there is no `ollama logs` subcommand# Linux (systemd)journalctl -u ollama --no-pager --follow --pager-end# macOS: cat ~/.ollama/logs/server.log — Docker: docker logs <container>
# Metrics in API response (durations in nanoseconds)curl -s http://localhost:11434/api/generate \ -d '{"model":"llama3.2","prompt":"test","stream":false}' \ | python3 -c "import sys, jsonr = json.load(sys.stdin)print(f'TTFT: {r[\"prompt_eval_duration\"]/1e9:.2f}s')print(f'Generation: {r[\"eval_duration\"]/1e9:.2f}s')print(f'Throughput: {r[\"eval_count\"]/(r[\"eval_duration\"]/1e9):.1f} tok/s')"8. Traces vs. metrics — OpenTelemetry for agent stacks
Prometheus and Grafana answer “Is the server healthy?”. They do not answer “Why did this request take 30 seconds?”.
The agent problem
In a RAG or multi-agent pipeline, one request crosses several services before producing an answer. If a user complains about 30s latency, Grafana may show the GPU at 70% utilization — not enough to diagnose the cause. You need the full request trace:
User request (30s total)├── VectorDB Qdrant — 4s (semantic search)├── Agent routing — 2s (tool selection)├── SearXNG web search — 20s (network timeout ⚠️)└── LLM generation — 4s (200 tokens @ 50 tok/s)Without traces, engineers search logs in each service separately. With OpenTelemetry traces, the full request appears as one waterfall in a dedicated tool.
OpenTelemetry (OTEL) — native vLLM and LiteLLM integration
LiteLLM and vLLM support OpenTelemetry natively10:
# vLLM — enable OTEL (export to local collector)vllm serve meta-llama/Llama-3.1-70B-Instruct \ --otlp-traces-endpoint http://localhost:4317# --collect-detailed-traces model for detailed traces (costly; all|model|worker)[^11]
# LiteLLM — litellm_config.yamllitellm_settings: callbacks: ["otel"]# and in the LiteLLM proxy environment:# OTEL_EXPORTER=otlp_grpc# OTEL_ENDPOINT=http://localhost:4317 # OTLP gRPCRecommended LLM observability stack
For a sovereign on-prem infrastructure, three options:
| Tool | Type | Strengths | Deployment |
|---|---|---|---|
| Langfuse (self-hosted, v4) | Traces + LLM eval | LLM-focused UI, cost per token, quality evaluation | Official Docker Compose (local use and testing); Helm in production11 |
| Jaeger | Distributed traces | CNCF standard, lightweight, Kubernetes-ready | Docker jaegertracing/all-in-one |
| Arize Phoenix | Traces + agent debug | Specialized for agents/RAG, free; Elastic License 2.0 (source-available, not OSI, forbids offering it as a managed service) | pip install arize-phoenix1213 |
Minimal recommended stack for a custodian agent:
# docker-compose.yml — add to existing stack# Langfuse (v4) is no longer a single container: it needs PostgreSQL, ClickHouse,# Redis/Valkey and S3 storage (MinIO locally). Start from the official docker-compose.yml# in the Langfuse repository (positioned as "local use and testing"); for# production, the Helm chart. See note [^6].# git clone https://github.com/langfuse/langfuse && cd langfuse && docker compose up -d
otel-collector: image: otel/opentelemetry-collector-contrib:latest volumes: - ./otel-config.yaml:/etc/otel-config.yaml command: ["--config=/etc/otel-config.yaml"] ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTPApplication state backup (DRP)
Monitoring detects failures — but without backups, recovery after an incident can take hours or lose data. Here is the state to protect in a local inference stack.
What to back up
| Component | Critical data | Recommended method |
|---|---|---|
| Qdrant (vector DB) | Collections + index snapshots | POST /collections/{name}/snapshots → local snapshot; copy to external storage |
| Milvus (vector DB) | Collections, segments, metadata | Milvus Backup CLI (milvus-backup create) to local S3 or NAS |
| SQLite (Memory Tree, histories) | .db file | Daily cp with rotation or hot sqlite3 .backup |
| vLLM / Ollama config | config.yaml, startup scripts | Versioned in Git — already covered |
| Models and adapters | GGUF weights, fine-tuned LoRA | Base weights = immutable (re-download); fine-tuned adapters = back up to NAS |
Suggested frequencies
Qdrant snapshot → daily (cron 02:00), 7-day retentionMilvus backup → daily (cron 03:00), 7-day retentionSQLite Memory Tree → hourly (if frequent writes), otherwise dailyLoRA adapters → after each training sessionQdrant snapshot example (curl)
# Create a snapshotcurl -X POST http://localhost:6333/collections/my_collection/snapshots
# List available snapshotscurl http://localhost:6333/collections/my_collection/snapshots
# Copy to external storage (mounted NAS example)cp /qdrant/snapshots/my_collection/*.snapshot /mnt/backup/qdrant/Recovery procedure (indicative RTO)
- Restart the container vLLM or Ollama → reload weights from SSD (~15–120 s depending on model)
- Restore Qdrant from latest snapshot →
PUT /collections/{name}/snapshots/recover - Restore SQLite → copy backup
.dbover the corrupted file - Verify health endpoint (
/healthor/v1/models)
See also
- ⚙️ Configure vLLM multi-GPU —
/metricsendpoint and parameters - 🖥️ Scenario C — Exo/Thunderbolt monitoring
- 🏭 Scenario D — datacenter RoCE monitoring
Sources and references
Footnotes
-
vLLM Project, Metrics (
/metricsendpoint, full list of exposed Prometheus metrics —vllm:kv_cache_usage_perc,vllm:num_preemptions,vllm:simple_kv_offload_*— metric deprecation policy), accessed 2026-10-09. https://docs.vllm.ai/en/stable/usage/metrics/ ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 -
Ollama, API Reference (
/api/generate,/api/chatendpoints andeval_count/eval_duration/prompt_eval_durationfields; no/metricsendpoint), accessed 2026-10-09. https://docs.ollama.com/api ↩ -
vLLM Project, Engine Arguments (
--gpu-memory-utilization,--max-model-len,--max-num-seqs), accessed 2026-10-09. https://docs.vllm.ai/en/stable/configuration/engine_args/ ↩ -
vLLM Project, Disaggregated Prefill and Decode (
--kv-transfer-config,OffloadingConnectorfor KV Cache CPU offload), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/disagg_prefill/ ↩ -
NVIDIA, DCGM Exporter (
etc/default-counters.csv: GPU_UTIL, MEM_COPY_UTIL, FB_USED, FB_FREE, GPU_TEMP, POWER_USAGE, NVLINK_BANDWIDTH_TOTAL in MB/s; FB_TOTAL absent from default counters), accessed 2026-10-09. https://github.com/NVIDIA/dcgm-exporter ↩ ↩2 -
Grafana Labs, NVIDIA DCGM Exporter Dashboard (ID 12239, GPU metrics visualization). https://grafana.com/grafana/dashboards/12239-nvidia-dcgm-exporter-dashboard/ ↩
-
Grafana Labs, Node Exporter Full Dashboard (ID 1860, system metrics — CPU, memory, disk, network). https://grafana.com/grafana/dashboards/1860-node-exporter-full/ ↩
-
vLLM Project,
vllm-project/vllmrepository,examples/observability/prometheus_grafana/(README,docker-compose.yaml,prometheus.yaml,grafana.json) andexamples/observability/dashboards/grafana/(performance_statistics.json,query_statistics.json),maintree accessed 2026-10-10. https://github.com/vllm-project/vllm/tree/main/examples/observability/prometheus_grafana ↩ -
Ollama, Troubleshooting (log locations:
journalctl -u ollama,~/.ollama/logs/server.log,docker logs), accessed 2026-10-09 ·ollama/ollamarepository,cmd/cmd.go(subcommand list, nologs). https://docs.ollama.com/troubleshooting · https://github.com/ollama/ollama ↩ -
LiteLLM, OpenTelemetry Integration (
litellm_settings.callbacks: ["otel"],OTEL_EXPORTER/OTEL_ENDPOINTvariables), accessed 2026-10-09 · vLLM Project, CLI Reference —vllm serve(--otlp-traces-endpoint,--collect-detailed-traces all|model|worker), accessed 2026-10-09. https://docs.litellm.ai/docs/observability/opentelemetry_integration · https://docs.vllm.ai/en/stable/cli/serve/ ↩ -
Langfuse, Self-Hosting (v4: PostgreSQL + ClickHouse + Redis/Valkey + S3 storage; Docker Compose for local use, Helm for production), accessed 2026-10-09. https://langfuse.com/self-hosting ↩
-
Arize AI, Arize Phoenix — LLM Observability (agent traces, RAG debugging, OTEL-compatible), accessed 2026-10-09. https://arize.com/docs/phoenix ↩
-
Arize AI,
Arize-ai/phoenixrepository,LICENSEfile (Elastic License 2.0) and Release arize-phoenix-v20.0.0, 2026-08-11. https://github.com/Arize-ai/phoenix/blob/main/LICENSE · https://github.com/Arize-ai/phoenix/releases/tag/arize-phoenix-v20.0.0 ↩