AI Mahjong Dojo Vol. 2: The Engine Decides, the LLM Verbalizes
When we left everything entirely to the LLM, it would sometimes get the specific numbers of mahjong wrong. To solve this, we implemented a division of roles: 'Programs handle calculations, LLMs handle explanations.'
- Do not leave numerical values or complex judgments to local LLMs
- Calculate accurate analysis data, such as the number of tiles to win (Akacho), using programs
- Prevent hallucinations by assigning the LLM only the task of 'verbalizing the analysis results'
- Improve response speed by optimizing inference model settings
Why you cannot leave everything to an LLM
When we first tried to have AI explain mahjong, the initial challenge was that local LLMs (large language models run on your own PC) tend to make mistakes regarding the specific numbers and judgments unique to mahjong.
While LLMs are skilled at generating text, they can sometimes provide answers that differ from facts in reasoning involving high-level calculations or precise numerical values. This is called hallucination (a phenomenon where AI tells plausible lies).
- Leaving everything to the LLM makes it easy for numerical errors to occur
- Accurate judgment requires strict analysis by a program
- If the explanation and the AI's thinking are inconsistent, it won't be effective for learning
A mechanism that separates 'Calculation' and 'Verbalization'
To solve this problem, we adopted a system in this tool where roles are clearly divided. Specifically, the program (engine) handles mahjong judgments and numerical calculations, and the LLM translates those results into 'words for beginners' to provide an explanation.
Accurate data such as the number of tiles to win (Akacho), the number of acceptable tiles, danger levels, and expected hands are all determined by program analysis. We designed it so that only these 'confirmed facts' are passed to the LLM to create the explanation.
- Data such as Akacho and acceptable tile counts are calculated by the program
- The LLM is given the role of creating beginner-friendly words based on those results
- Since the CPU (computer) also plays using the same analysis data, it prevents inconsistencies between the explanation and the thinking
Efforts to reduce hallucinations
During development, by improving the system to pass facts calculated by the engine to the LLM, we were able to significantly reduce errors caused by fabricated stories.
For example, there were previously bugs where 'the CPU won (Agari), but the LLM congratulated the player side.' Currently, these inconsistencies are corrected through this division of roles.
- Suppress hallucinations by only passing facts to the LLM
- Ensure consistency between the engine's actions and the explanations
For even more comfortable operation
Additionally, we are reviewing model settings for a smoother experience. When using 'thinking' (inference) models such as gemma4 or Qwen3, it has been confirmed that response speeds significantly increase by setting 'Disable thinking mode' in the settings.
In this way, understanding the characteristics of AI, assigning appropriate roles, and combining optimal settings are key points for creating a practical tool.
- Speed up by disabling the 'thinking mode' of inference models
- Achieve both accuracy and speed through division of roles
- Those who want to create their own tools using AI
- Those interested in mechanisms to improve LLM accuracy
- Those who want to know how to utilize local AI
- Local LLM
- A large language model that runs on your own PC or server without using cloud services (such as ChatGPT).
- Hallucination
- The AI telling plausible lies. It refers to the phenomenon of generating information not based on facts.
- Akacho (Number of tiles to win)
- A numerical value in mahjong indicating how many more tiles need to be drawn to reach a winning state.
FAQ
Why don't you let the LLM do all the calculations?
While LLMs excel at text generation, if they are left to handle all rigorous numerical calculations and probability judgments like mahjong, they often present incorrect values (hallucinations).
Will accuracy drop if thinking mode is disabled?
According to official information, disabling thinking mode in inference models significantly speeds up responses. Since 'calculations' are performed by the program through division of roles, we can take advantage of this speed benefit for the verbalization of explanations.
In AI Mahjong Dojo, to compensate for the LLM's weakness in numerical accuracy, we use a system where analysis results from a program are passed to the LLM for verbalization. This achieves easy-to-understand explanations for beginners while suppressing hallucinations. Furthermore, by optimizing inference model settings, we pursue even more comfortable response speeds.