CUDA
๐ Short definition
NVIDIAโs software platform that lets programs use NVIDIA GPUs for computation, and therefore run AI models on these cards.
๐ Detailed definition
NVIDIA defines it as โa parallel computing platform and programming modelโ that speeds up computation by harnessing the GPU1. It includes the driver, the compiler and optimized libraries on which frameworks (PyTorch) and inference engines (vLLM, SGLang, TensorRT-LLM) rely.
CUDA only runs on NVIDIA hardware. Other vendors have their own stack:
| Vendor | Software stack | Role |
|---|---|---|
| NVIDIA | CUDA | reference of the AI ecosystem1 |
| AMD | ROCm | โthe AMD open software stack for GPU-accelerated computingโ2 |
| Apple | Metal, MLX | MLX is โan array framework for machine learning on Apple siliconโ3 |
๐ก Why it matters for on-premise AI
Something new (model, format, optimization) often lands on CUDA first. Choosing an NVIDIA machine simplifies installing the engines; choosing AMD or Apple means checking that the target engine and model are supported on ROCm or MLX.
โ ๏ธ Common pitfalls
- Assuming a โGPUโ tool works on any GPU: many only support CUDA, or support ROCm and Metal with a delay.
- Ignoring the version: an engine often requires a minimum CUDA and driver version.
๐ To go deeper
- ๐ง APU & Unified Memory (CUDA, ROCm and MLX depending on the machine)
๐ See also
Footnotes
-
NVIDIA, CUDA Programming Guide (โCUDA is a parallel computing platform and programming model developed by NVIDIAโฆ by harnessing the power of the GPUโ), accessed 2026-10-11. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html โฉ โฉ2
-
AMD, What is ROCm? (โROCm is the AMD open software stack for GPU-accelerated computingโ), accessed 2026-10-11. https://rocm.docs.amd.com/en/latest/what-is-rocm.html โฉ
-
Apple ml-explore, MLX โ GitHub repository (โMLX is an array framework for machine learning on Apple siliconโ, unified memory model), accessed 2026-10-11. https://github.com/ml-explore/mlx โฉ