Vulkan vs CUDA for local models
Short answer
CUDA is faster on NVIDIA hardware — about 16% on a 7B model in our measurements — and is roughly 611 MB and NVIDIA-only. Vulkan is about 32 MB, works on AMD and Intel too, and was about 4% faster on a small 1.5B model.
The measurements
On generation with a 7B model, Vulkan came out about 16% slower than CUDA. On a 1.5B model it was about 4% faster, which is the kind of result that tells you the overhead profile differs rather than that one is simply better.
Runtime size is the starker difference: roughly 32 MB against 611 MB.
Why Nightjar defaults to Vulkan
A browser cannot assume an NVIDIA card. Vulkan runs on AMD, Intel and NVIDIA from one build, which means the install works on whatever is in the machine rather than on a subset.
Adding 611 MB to a browser download to gain 16% on the largest models, for one vendor's customers only, is the wrong trade for this product.
When CUDA would be the right call
A dedicated inference setup on known NVIDIA hardware, running large models continuously, where that 16% compounds. That is a different product from a browser.
The caveat
One machine, one driver version, two model sizes. Treat these as the direction of the difference rather than as a precise ratio, and measure your own if it matters.
Other ways people ask this
- vulkan vs cuda llama cpp
- Why would a browser ship Vulkan instead of CUDA?