Is slow AI response caused by insufficient VRAM? The true identity of speed differences revealed by actual measurements
When running AI on your own PC, do you ever find that the answers aren't coming out as quickly as you expected? In fact, the cause of this could often lie in an insufficient amount of 'VRAM (Video RAM)'. This time, we will explain the results of our verification on how much the speed changes depending on where the model data is being loaded.
- If a model does not fit into the VRAM, the majority of the processing is shifted to the slow CPU, causing an extreme drop in speed.
- In actual measurements, a speed difference of more than approximately 9 times occurred between the state where the model fit in the VRAM and the state where it ran on the CPU only.
- When using tools like Ollama, you can check the VRAM loading status with specific commands.
What is VRAM? Why does it affect speed?
When running AI, the model data is basically loaded into 'VRAM (Video RAM)', which is dedicated memory mounted on the GPU (the part that performs image processing, etc.). Because the GPU has very high calculation speeds, the AI operates smoothly when all the data is contained within the VRAM.
However, if the model size exceeds the VRAM capacity, the overflowing data spills over into the main memory (memory shared with the CPU). At this point, because the CPU takes over much of the processing, the speed drops dramatically.
- State where it fits in VRAM: The GPU is fully utilized, and a high-speed response is obtained.
- State where it overflows from VRAM: Since both the CPU and main memory are used together, processing stalls and wait times become long.
[Actual Measurement] Surprising speed difference based on VRAM availability
I compared the speeds when actually running models using this account's PC. The model used for verification is gemma4:12b (Q8, memory usage 14.2GB).
The values when running this model on an 'RTX 4080' graphics board equipped with 16GB of VRAM are as follows.
- State where the model fit in VRAM: 43.5 tokens/sec (feels like it comes out instantly)
- State where it ran on CPU only: 4.6 tokens/sec (feels like the wait time is noticeable)
What we can say from the results showing an approximately 9x difference
Looking at the actual measurement results above, you can see that there is a speed difference of about 9 times or more depending on whether it is in the VRAM or not. While operation is possible with just the CPU, if you are seeking practical comfort, it is extremely important to fit the model size into the VRAM capacity.
When running AI on your own PC, I recommend checking the following three points as a checklist.
- Does the model size exceed the VRAM capacity of your GPU?
- Is it set to use both the CPU and main memory together?
- Can you correctly check the VRAM loading status using the tool you are using?
How to check in Ollama
If you are using 'Ollama', one of the tools for running AI, there is a way to check whether the model is correctly loaded into the VRAM.
According to official information, you can grasp the situation by checking the `/api/ps` endpoint. If the `size_vram` and `size` values displayed here match, it is proof that all the model data is loaded into the VRAM. If these values do not match, it is highly likely that the speed is dropping due to a lack of memory.
- If they match: A state where the power of the GPU is being maximized.
- If they do not match: There is a possibility that it has switched to slow processing due to lack of memory.
- People who want to run AI on their own PC but feel that the operation is slow for some reason
- People who want to know the compatibility between GPU specs and AI models
- People who are running models with Ollama but want to know the detailed behavior
- VRAM (Video RAM)
- Dedicated memory mounted on the GPU (graphics board). Model data is placed here to accelerate AI calculations.
- tokens/sec
- The unit of characters or words that AI generates per second. The higher this value, the faster the AI's response is displayed.
- Ollama
- Software for easily running AI models on your own PC (local environment).
FAQ
If VRAM is insufficient, can I solve it by making the model smaller?
Yes. By reducing the number of model parameters (e.g., 12b) or using quantized versions (a technology to compress models), it becomes easier to fit into the VRAM, and speed may improve.
Are there situations where it's better to run on CPU only?
While the speed is very slow, if you absolutely want to run a large model on a PC with extremely low VRAM, running on the CPU is one option. However, if you seek practical speed, loading into VRAM is recommended.
The biggest key to running AI comfortably in a local environment is how much of the model data you can fit into the VRAM. Actual measurements confirmed that a speed difference of about 9 times can occur based on the presence of VRAM. Checking your PC's VRAM capacity and choosing a model size that fits accordingly is the first step toward utilizing AI without stress.