Skip to content

Running a local LLM without a graphics card

Short answer

It works, and model choice decides whether it is usable. On CPU alone, Qwen 3.5 2B generated 15 tokens per second on our test machine while Gemma 4 12B managed 2.8.

The numbers

From our own measurements with no GPU offload at all: Qwen 3.5 2B at 15 tokens per second, Gemma 4 E2B at 13, Qwen 3.5 9B at 3.2, Gemma 4 12B at 2.8.

Fifteen tokens per second is about eleven words a second, which reads faster than most people do. Three is a wait.

What to run

Stay small. The useful insight is that small does not mean incapable here: Qwen 3.5 2B scored 16 out of 16 on our reading evaluation, matching the 12B model at a fifth of the size.

For reading a document in front of it — which is the job — a well-trained small model is close to sufficient.

What helps

Memory bandwidth more than core count. Generation is memory-bound, so dual-channel RAM and a modern platform matter more than extra cores.

Shorter conversations. The cache grows with history, and on CPU that is felt.

Patience with the first answer: time to first token is worse on CPU, especially on a long document.

Other ways people ask this

  • run llm without gpu
  • local llm cpu only speed

Related