Skip to content

Vulkan vs CUDA for local models

Short answer

CUDA is faster on NVIDIA hardware — about 16% on a 7B model in our measurements — and is roughly 611 MB and NVIDIA-only. Vulkan is about 32 MB, works on AMD and Intel too, and was about 4% faster on a small 1.5B model.

The measurements

On generation with a 7B model, Vulkan came out about 16% slower than CUDA. On a 1.5B model it was about 4% faster, which is the kind of result that tells you the overhead profile differs rather than that one is simply better.

Runtime size is the starker difference: roughly 32 MB against 611 MB.

Why Nightjar defaults to Vulkan

A browser cannot assume an NVIDIA card. Vulkan runs on AMD, Intel and NVIDIA from one build, which means the install works on whatever is in the machine rather than on a subset.

Adding 611 MB to a browser download to gain 16% on the largest models, for one vendor's customers only, is the wrong trade for this product.

When CUDA would be the right call

A dedicated inference setup on known NVIDIA hardware, running large models continuously, where that 16% compounds. That is a different product from a browser.

The caveat

One machine, one driver version, two model sizes. Treat these as the direction of the difference rather than as a precise ratio, and measure your own if it matters.

Other ways people ask this

  • vulkan vs cuda llama cpp
  • Why would a browser ship Vulkan instead of CUDA?

Related