πΌοΈ Multimodality: Hardware Impact (VRAM & KV Cache)
Why multimodality changes the hardware equation
A pure LLM processes text: each token is a numerical vector from a vocabulary. Memory complexity is predictable and well-documented.
A VLM (Vision Language Model) adds an upstream component: a visual encoder that transforms image pixels into a sequence of vectors the LLM can βreadβ. This additional path has direct consequences for VRAM, the KV Cache, and latency.
The visual encoder: anatomy and VRAM footprint
How it works
flowchart TD
A["πΌοΈ Image (pixels)"] --> B["Visual encoder\n(e.g. CLIP-ViT-L/14, SigLIP)\nsplits into 14Γ14 px patches\nencodes each patch as a vector"]
B --> C["Visual token sequence\n(256 to a few thousand tokens depending on the model and resolution)"]
C --> D["Projector (MLP bridge)"]
D --> E["LLM backbone\n(Qwen, Mistral, LLaMAβ¦)"]
The encoder and projector are additional weights loaded on top of the LLM backbone.
Approximate VRAM footprint by model family
| VLM model | LLM backbone | Visual encoder | Total VRAM (FP16) | Visual tokens / image |
|---|---|---|---|---|
| LLaVA 1.6 (Mistral 7B) | 7B | CLIP-ViT-L/14, 304M (~0.6 GB) | ~15 GB | 576 (base 336Γ336 view) to ~2,880 (AnyRes: 4 tiles + global view) |
| Qwen2-VL 7B | 7B | ViT 675M (~1.35 GB) | ~16 GB | 4β16,384 (dynamic; ~1,369 for a 1024Γ1024 image) |
| Qwen2-VL 72B | 72B | ViT 675M (~1.35 GB) | ~144 GB | 4β16,384 (dynamic; ~1,369 for a 1024Γ1024 image) |
| Pixtral 12B (Mistral) | 12B | Vision encoder 400M (~0.8 GB) | ~26 GB | up to 1,024 |
| Gemma 3 27B Vision | 27B | SigLIP 400M (~0.8 GB) | ~54 GB | 256 per 896Γ896 image (+ 256 per crop with Pan & Scan) |
Sources: LLaVA 1 Β· Qwen2-VL 2
Impact on the KV Cache
The KV Cache stores intermediate states for each token in the context. Visual tokens behave exactly like text tokens: they occupy the same amount of space per token.
Reference formula
For an FP16 model:
KV Cache per token = 2 (K+V) Γ num_layers Γ KV_heads Γ head_dim Γ 2 bytesFor a typical 7B LLM with GQA (Mistral 7B, the LLaVA 1.6 backbone: 32 layers, 8 KV heads, 128 head dim):
β 2 Γ 32 Γ 8 Γ 128 Γ 2 = 131,072 bytes β 128 KiB (~0.13 MB) per token β 512 KiB with full MHA (32 KV heads, e.g. Llama 2 7B)Text vs image comparison
| Input | Typical tokens | Estimated KV Cache (7B GQA, ~0.13 MB/token) |
|---|---|---|
| 500-word text document | ~650 tokens | ~85 MB |
| 336Γ336 image (LLaVA 1.5, or LLaVA 1.6 base view) | ~576 tokens | ~75 MB |
| 1024Γ1024 image (LLaVA 1.6 AnyRes: 4 tiles + global view) | ~2,880 tokens | ~377 MB |
| 1024Γ1024 image (Qwen2-VL dynamic) | ~1,369 tokens | ~179 MB (~79 MB on the Qwen2-VL 7B backbone: 28 layers, 4 KV heads) |
| 10-page PDF converted to images | ~5,000β14,000 tokens (β 500 to 1,400 per page depending on the model and resolution) | ~0.65β1.8 GB |
Source: vLLM metrics 3
Audio: Whisper as pre-processing
Audio transcription (meetings, dictation, client calls) is often mentioned alongside vision, but its architecture is fundamentally different.
Whisper is not a VLM. It is an independent sequence-to-sequence model:
flowchart TD
A["π΅ Audio (WAV/MP3)"] --> B["Whisper\n(separate model β 39 MB to 1.5 GB)\ntranscribes to text"]
B --> C["Text (normal tokens)"]
C --> D["LLM backbone\n(if transcript analysis is needed)"]
Whisper VRAM footprint
| Whisper model | Parameters | VRAM |
|---|---|---|
| tiny | 39 M | ~80 MB |
| base | 74 M | ~145 MB |
| small | 244 M | ~480 MB |
| medium | 769 M | ~1.5 GB |
| large-v3 | 1.5 B | ~3 GB |
Source: OpenAI Whisper 4
Real time: an alternative to Whisper. To caption a live meeting, Mistral has published Voxtral Mini 4B Realtime (Apache 2.0) since February 2026: a natively streaming transcription model with an adjustable delay from 80 ms to 2.4 s, 13 languages including French (6.4% WER in French at a 480 ms delay according to the publisher), served by vLLM5. Unlike Whisper, it does not sit on a CPU: the publisher indicates a GPU of at least 16 GB (BF16 weights, ~4.4B parameters), to plan on top of the LLM. Whisper on CPU remains the right choice for batch; Voxtral is justified when latency matters. Mistral releases per-language variants (Arabic dialects in October 2026).
Architectural consequence: Whisper can coexist with a VLM on the same machine without significant VRAM competition, provided they are not run simultaneously on the GPU. In an asynchronous pipeline (transcription β LLM summary), a sequential approach works perfectly on Blueprint B.
Blueprint map
| Blueprint | VLM feasible? | Recommended model | Main constraint |
|---|---|---|---|
| A β Dev Lab (RTX 4090, 24 GB) | β Yes | LLaVA 1.6 7B, Qwen2-VL 7B | HD images limited in simultaneous batch |
| A β Dev Lab (64 GB unified memory) | β Yes | Qwen2-VL 7B or Pixtral 12B | Unified memory bandwidth is the limiting factor |
| B β SME Appliance (128 GB unified memory) | β Yes | Qwen2-VL 7B or 72B Q4 | 70B with vision is slow but functional |
| C β Desktop Cluster (4Γ 64 GB unified) | β Yes | Pixtral 12B, Qwen2-VL 72B distributed | Exo required to distribute a 72B vision model |
| D β Datacenter (8Γ 80 GB GPU) | β Yes | Qwen2-VL 72B, Pixtral Large | High-concurrency production use case |
Production watchpoints
-
Context length surprises: a βsimpleβ request (image + short question) can consume up to ~3,000 tokens (LLaVA 1.6 with AnyRes), where the equivalent text-only question used only 50. Adjust
max_model_lenin vLLM accordingly. -
Asymmetric prefill: prefilling an image (passing all visual tokens through the transformer) is computationally denser than an equivalent text prefill in tokens. Time-to-first-token (TTFT) is higher.
-
Resolution and tiling: modern VLMs (LLaVA 1.6 HD, Qwen2-VL) dynamically adapt the number of visual tokens to image resolution. Sending unresized images can multiply cost by 4β8Γ.
-
Partial quantisation: visual encoders handle aggressive quantisation poorly (Q4 or Q2). If you quantise the backbone to save VRAM, keep the visual encoder in FP16 or Q8.
See also
- KV Cache β detailed attention cache mechanics
- VRAM β model memory footprint
- ποΈ Quantisation 4-bit & 8-bit β reducing VRAM footprint
- π§ KV Cache and context window
- πΎ Unified memory vs RAM vs VRAM
- π’ Scenario B β SME Appliance
Sources and references
Footnotes
-
Liu et al., Visual Instruction Tuning (LLaVA) (architecture: CLIP-ViT-L/14 encoder + MLP projector + LLM backbone; encoder VRAM derived from published parameter count). https://arxiv.org/abs/2304.08485 β©
-
Alibaba Cloud, Qwen2-VL model documentation (dynamic visual tokens by resolution: 4 to 16,384 per image by default, sides rounded to the nearest multiple of 28, accessed 2026-11-19); P. Wang et al., Qwen2-VL (arXiv:2409.12191: ViT encoder of about 675 M parameters). https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct β©
-
vLLM Project, Production Metrics β KV cache usage (KV Cache behaviour under continuous batching). https://docs.vllm.ai/en/stable/usage/metrics/ β©
-
OpenAI, Whisper model card (tiny to large-v3 sizes, parameters and indicative memory footprint). https://github.com/openai/whisper β©
-
Mistral AI, Voxtral Mini 4B Realtime 2602 (real-time streaming transcription, 80 ms β 2.4 s delay, 480 ms recommended; 13 languages; BF16, one GPU β₯ 16 GB; vLLM; Apache 2.0), Hugging Face, February 2026; Voxtral Mini 4B Realtime Arabic, 2026-10-08. https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 Β· https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-Arabic β©