Very high-bandwidth memory attached to the GPU, storing weights, KV Cache and buffers; its capacity and bandwidth determine which models run without offloading.Standard bus connecting CPU, GPU, SSD and other peripherals; it limits CPU-to-GPU offloading and Tensor Parallelism on multi-GPU setups without NVLink.
A guide by Damien BECHERINI, written with the help of AI agents. The author reviews and approves every change before publication, and every figure cites its primary source. Content dedicated to the public domain (CC0 1.0). Β· Method Β· Licence