Skip to content

πŸ–ΌοΈ Multimodality: Hardware Impact (VRAM & KV Cache)


Why multimodality changes the hardware equation

A pure LLM processes text: each token is a numerical vector from a vocabulary. Memory complexity is predictable and well-documented.

A VLM (Vision Language Model) adds an upstream component: a visual encoder that transforms image pixels into a sequence of vectors the LLM can β€œread”. This additional path has direct consequences for VRAM, the KV Cache, and latency.


The visual encoder: anatomy and VRAM footprint

How it works

flowchart TD
    A["πŸ–ΌοΈ Image (pixels)"] --> B["Visual encoder\n(e.g. CLIP-ViT-L/14, SigLIP)\nsplits into 14Γ—14 px patches\nencodes each patch as a vector"]
    B --> C["Visual token sequence\n(256 to a few thousand tokens depending on the model and resolution)"]
    C --> D["Projector (MLP bridge)"]
    D --> E["LLM backbone\n(Qwen, Mistral, LLaMA…)"]

The encoder and projector are additional weights loaded on top of the LLM backbone.

Approximate VRAM footprint by model family

VLM modelLLM backboneVisual encoderTotal VRAM (FP16)Visual tokens / image
LLaVA 1.6 (Mistral 7B)7BCLIP-ViT-L/14, 304M (~0.6 GB)~15 GB576 (base 336Γ—336 view) to ~2,880 (AnyRes: 4 tiles + global view)
Qwen2-VL 7B7BViT 675M (~1.35 GB)~16 GB4–16,384 (dynamic; ~1,369 for a 1024Γ—1024 image)
Qwen2-VL 72B72BViT 675M (~1.35 GB)~144 GB4–16,384 (dynamic; ~1,369 for a 1024Γ—1024 image)
Pixtral 12B (Mistral)12BVision encoder 400M (~0.8 GB)~26 GBup to 1,024
Gemma 3 27B Vision27BSigLIP 400M (~0.8 GB)~54 GB256 per 896Γ—896 image (+ 256 per crop with Pan & Scan)

Sources: LLaVA 1 Β· Qwen2-VL 2


Impact on the KV Cache

The KV Cache stores intermediate states for each token in the context. Visual tokens behave exactly like text tokens: they occupy the same amount of space per token.

Reference formula

For an FP16 model:

KV Cache per token = 2 (K+V) Γ— num_layers Γ— KV_heads Γ— head_dim Γ— 2 bytes

For a typical 7B LLM with GQA (Mistral 7B, the LLaVA 1.6 backbone: 32 layers, 8 KV heads, 128 head dim):

β‰ˆ 2 Γ— 32 Γ— 8 Γ— 128 Γ— 2 = 131,072 bytes β‰ˆ 128 KiB (~0.13 MB) per token β€” 512 KiB with full MHA (32 KV heads, e.g. Llama 2 7B)

Text vs image comparison

InputTypical tokensEstimated KV Cache (7B GQA, ~0.13 MB/token)
500-word text document~650 tokens~85 MB
336Γ—336 image (LLaVA 1.5, or LLaVA 1.6 base view)~576 tokens~75 MB
1024Γ—1024 image (LLaVA 1.6 AnyRes: 4 tiles + global view)~2,880 tokens~377 MB
1024Γ—1024 image (Qwen2-VL dynamic)~1,369 tokens~179 MB (~79 MB on the Qwen2-VL 7B backbone: 28 layers, 4 KV heads)
10-page PDF converted to images~5,000–14,000 tokens (β‰ˆ 500 to 1,400 per page depending on the model and resolution)~0.65–1.8 GB

Source: vLLM metrics 3


Audio: Whisper as pre-processing

Audio transcription (meetings, dictation, client calls) is often mentioned alongside vision, but its architecture is fundamentally different.

Whisper is not a VLM. It is an independent sequence-to-sequence model:

flowchart TD
    A["🎡 Audio (WAV/MP3)"] --> B["Whisper\n(separate model β€” 39 MB to 1.5 GB)\ntranscribes to text"]
    B --> C["Text (normal tokens)"]
    C --> D["LLM backbone\n(if transcript analysis is needed)"]

Whisper VRAM footprint

Whisper modelParametersVRAM
tiny39 M~80 MB
base74 M~145 MB
small244 M~480 MB
medium769 M~1.5 GB
large-v31.5 B~3 GB

Source: OpenAI Whisper 4

Real time: an alternative to Whisper. To caption a live meeting, Mistral has published Voxtral Mini 4B Realtime (Apache 2.0) since February 2026: a natively streaming transcription model with an adjustable delay from 80 ms to 2.4 s, 13 languages including French (6.4% WER in French at a 480 ms delay according to the publisher), served by vLLM5. Unlike Whisper, it does not sit on a CPU: the publisher indicates a GPU of at least 16 GB (BF16 weights, ~4.4B parameters), to plan on top of the LLM. Whisper on CPU remains the right choice for batch; Voxtral is justified when latency matters. Mistral releases per-language variants (Arabic dialects in October 2026).

Architectural consequence: Whisper can coexist with a VLM on the same machine without significant VRAM competition, provided they are not run simultaneously on the GPU. In an asynchronous pipeline (transcription β†’ LLM summary), a sequential approach works perfectly on Blueprint B.


Blueprint map

BlueprintVLM feasible?Recommended modelMain constraint
A β€” Dev Lab (RTX 4090, 24 GB)βœ… YesLLaVA 1.6 7B, Qwen2-VL 7BHD images limited in simultaneous batch
A β€” Dev Lab (64 GB unified memory)βœ… YesQwen2-VL 7B or Pixtral 12BUnified memory bandwidth is the limiting factor
B β€” SME Appliance (128 GB unified memory)βœ… YesQwen2-VL 7B or 72B Q470B with vision is slow but functional
C β€” Desktop Cluster (4Γ— 64 GB unified)βœ… YesPixtral 12B, Qwen2-VL 72B distributedExo required to distribute a 72B vision model
D β€” Datacenter (8Γ— 80 GB GPU)βœ… YesQwen2-VL 72B, Pixtral LargeHigh-concurrency production use case

Production watchpoints

  1. Context length surprises: a β€œsimple” request (image + short question) can consume up to ~3,000 tokens (LLaVA 1.6 with AnyRes), where the equivalent text-only question used only 50. Adjust max_model_len in vLLM accordingly.

  2. Asymmetric prefill: prefilling an image (passing all visual tokens through the transformer) is computationally denser than an equivalent text prefill in tokens. Time-to-first-token (TTFT) is higher.

  3. Resolution and tiling: modern VLMs (LLaVA 1.6 HD, Qwen2-VL) dynamically adapt the number of visual tokens to image resolution. Sending unresized images can multiply cost by 4–8Γ—.

  4. Partial quantisation: visual encoders handle aggressive quantisation poorly (Q4 or Q2). If you quantise the backbone to save VRAM, keep the visual encoder in FP16 or Q8.


See also


Sources and references

Footnotes

  1. Liu et al., Visual Instruction Tuning (LLaVA) (architecture: CLIP-ViT-L/14 encoder + MLP projector + LLM backbone; encoder VRAM derived from published parameter count). https://arxiv.org/abs/2304.08485 ↩

  2. Alibaba Cloud, Qwen2-VL model documentation (dynamic visual tokens by resolution: 4 to 16,384 per image by default, sides rounded to the nearest multiple of 28, accessed 2026-11-19); P. Wang et al., Qwen2-VL (arXiv:2409.12191: ViT encoder of about 675 M parameters). https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct ↩

  3. vLLM Project, Production Metrics β€” KV cache usage (KV Cache behaviour under continuous batching). https://docs.vllm.ai/en/stable/usage/metrics/ ↩

  4. OpenAI, Whisper model card (tiny to large-v3 sizes, parameters and indicative memory footprint). https://github.com/openai/whisper ↩

  5. Mistral AI, Voxtral Mini 4B Realtime 2602 (real-time streaming transcription, 80 ms – 2.4 s delay, 480 ms recommended; 13 languages; BF16, one GPU β‰₯ 16 GB; vLLM; Apache 2.0), Hugging Face, February 2026; Voxtral Mini 4B Realtime Arabic, 2026-10-08. https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 Β· https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-Arabic ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.