Running a local LLM without a graphics card
Short answer
It works, and model choice decides whether it is usable. On CPU alone, Qwen 3.5 2B generated 15 tokens per second on our test machine while Gemma 4 12B managed 2.8.
The numbers
From our own measurements with no GPU offload at all: Qwen 3.5 2B at 15 tokens per second, Gemma 4 E2B at 13, Qwen 3.5 9B at 3.2, Gemma 4 12B at 2.8.
Fifteen tokens per second is about eleven words a second, which reads faster than most people do. Three is a wait.
What to run
Stay small. The useful insight is that small does not mean incapable here: Qwen 3.5 2B scored 16 out of 16 on our reading evaluation, matching the 12B model at a fifth of the size.
For reading a document in front of it — which is the job — a well-trained small model is close to sufficient.
What helps
Memory bandwidth more than core count. Generation is memory-bound, so dual-channel RAM and a modern platform matter more than extra cores.
Shorter conversations. The cache grows with history, and on CPU that is felt.
Patience with the first answer: time to first token is worse on CPU, especially on a long document.
Other ways people ask this
- run llm without gpu
- local llm cpu only speed
Related
- Up a levelGuides: local AI, browser privacy and the hardware to run it
- Closely relatedRunning a local model on integrated graphics
- Closely relatedWhat a local model does to your battery
- Closely relatedWhat a local model actually costs to run
- Closely relatedCan you run a local LLM on an NPU?
- Elsewhere on the siteCan your PC run a local AI model?
- Elsewhere on the siteHardware guide for local AI