Skip to content

NPU

📝 Short definition

Hardware accelerator specialized for neural-network ops, integrated in most modern SoCs (Apple Neural Engine, AMD XDNA, Qualcomm Hexagon, Intel NPU).

📖 Detailed definition

An NPU is optimized for low-precision matrix ops (INT8, INT4) with very low power draw. It excels at light, repetitive tasks: speech recognition, image classification, short translation, keyword spotting.

For large LLMs (7B+), NPUs are usually unused by current inference engines for two reasons:

  1. Insufficient memory capacity: the NPU accesses system memory on channels often more limited than the iGPU.
  2. Weak software support: Ollama, llama.cpp, and vLLM do not route to NPUs — they use Metal (Apple), ROCm (AMD), or CUDA (NVIDIA).
NPUSoCTOPS (INT8)Used for LLM?
Apple Neural EngineM4 Max~38 TOPSNo (Ollama/MLX → Metal GPU)
AMD XDNA 2Gorgon Halo~50 TOPSNo (ROCm/Vulkan → iGPU)
Qualcomm HexagonSnapdragon X Elite~45 TOPSPartially (Copilot+ on Windows)

💡 Why it matters for on-prem AI

The NPU matters for edge and embedded tasks: local transcription, data filtering, video streams. For an on-prem LLM assistant appliance, the NPU is transparent — the iGPU does the work.

⚠️ Common pitfalls

  • Buying an “AI PC” on NPU TOPS for LLM inference: GPU memory bandwidth matters, not NPU TOPS.
  • Confusing vendor NPU benchmarks (INT4-tuned tasks) with general LLM performance.

📚 Go deeper

🔗 See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.