2026-11-20Translated from the Japanese original

Why AI Generation Speed Doesn't Change Even When Reducing CPU Thread Count

When thinking about running AI on your own PC, have you ever wondered how much the CPU specs or thread count actually matter? Generally, it is assumed that the more processing power you use, the faster it gets, but actual measurements can sometimes yield surprising results. In this article, I will explain the actual measurement results showing that generation speed hardly changed even when limiting CPU threads, as well as the underlying mechanisms.

Key points
  • In actual measurements, the impact on generation speed was extremely small even when limiting the number of CPU threads.
  • AI text generation processes tend to be more influenced by memory data transfer speeds than by CPU calculation power.
  • Adjusting the number of threads to suit your PC environment can be one possible option.

Comparison via Actual Measurement Data: Results of 8 Cores vs. 4 Threads

I compared the generation speeds when running an AI model using only the CPU in a specific PC environment. For the comparison conditions, I performed measurements by limiting the CPU's processing power to two patterns: '8 cores' and '4 threads'.

The models used for measurement and their respective speeds (tokens/sec) are as follows.

First, with the gemma4:12b model, the result was 4.6 tokens/sec at 8 cores and 4.6 tokens/sec when limited to 4 threads. This value is at a speed level where 'the wait time is noticeable,' indicating that no change in speed was observed even when the number of threads was halved.

Next, with the qwen3.5:9b model, the results were 11.2 tokens/sec at 8 cores and 11.0 tokens/sec at 4 threads. This is a 'comfortable' speed that is faster than reading, and again, the speed drop was minimal even when narrowing down the threads.

  • gemma4:12b (8 cores): 4.6 tokens/sec (wait time is noticeable)
  • gemma4:12b (4 threads): 4.6 tokens/sec (wait time is noticeable)
  • qwen3.5:9b (8 cores): 11.2 tokens/sec (comfortable)
  • qwen3.5:9b (4 threads): 11.0 tokens/sec (comfortable)

Why Speed Doesn't Change Even When Reducing Thread Count

Why did the speed not change despite limiting the CPU's processing power? The reason lies in the nature of the processing when an AI generates text.

The process of an AI model generating the next word is not determined solely by high-level 'calculation speed.' Rather, it is said to be heavily influenced by 'data transfer speed from memory'—how quickly necessary data can be read from memory (the location where data is temporarily stored).

The results of this actual measurement suggest that this 'memory transfer speed' may be the bottleneck (the point causing processing delays) in the overall process. In other words, no matter how high you set the CPU's calculation power, if the data supply from the memory cannot keep up, the generation speed will hit a plateau.

  • AI generation processes tend to depend more on data readout speeds from memory than on CPU calculation speeds.
  • When memory transfer speed becomes the bottleneck, increasing the number of threads will not result in a speed improvement.

Tips for Adjusting to Your Own PC Environment

From this verification, we learned that utilizing all CPU threads does not necessarily translate directly into faster AI generation speeds. You might not need to hesitate to adopt AI because you are overly worried about specs.

When running AI in your PC environment, one approach is to try adjusting by intentionally limiting the number of threads to make parallel processing with other tasks smoother. Since official information suggests that AI processing is prone to being influenced by memory speed, I recommend first trying to find the optimal balance for yourself while adjusting the thread count in your current environment.

  • It is important to try things in your own environment first without worrying excessively about insufficient specs.
  • By limiting the number of threads, adjustments can be made to prioritize coexistence with other applications.
Useful for
  • Beginners who want to run AI on their own PC
  • Those who want to know the relationship between PC specs and AI speed
Glossary
Thread
The unit of work that a CPU can process at one time.
Token
The minimum unit when an AI processes words.
Bottleneck
The slowest part that limits the processing speed of the entire system.

FAQ

Does the speed really not change even if I reduce the number of threads?

In this actual measurement, because the memory transfer speed was the bottleneck, the result was that the speed remained almost the same even when reducing the number of threads. However, there is a possibility that changes may occur depending on the model and PC configuration.

How can I increase the speed?

According to the sources used in this test, data readout speed from memory is more important than calculation speed. Therefore, it is considered highly likely that memory performance and speed will have a greater impact than increasing the number of CPU threads.

Summary

Because AI text generation is more influenced by memory readout speed than CPU calculation power, the speed may not drop even if the number of threads is reduced. Adjusting the number of threads to suit your environment is also an effective method.

This article was translated from Japanese by AI; numbers and model names were automatically checked against the original. The original was written with AI from the sources cited there.