2026-10-14Translated from the Japanese original

[Series Finale] Tips for Making AI Remember Long Stories and Criteria for Choosing Models Without Failing

When weaving long stories with AI, the biggest wall is the 'limits of memory.' As conversations with AI grow longer, the previous content often overflows from the memory frame (context), causing the AI to forget important settings and promises. To solve this problem, the 'Narratable' tool I developed this time introduces a mechanism that automatically summarizes old statements into a 'synopsis' before the conversation accumulates too much. By doing this, important information—such as who did what and what promises were made—continues to remain, making it possible to progress a long story while maintaining consistency.

Key points
  • Fixing Output via JSON Schema Internally, the system uses a JSON schema (rules defining the structure of data) to fix the AI's output into four categories: 'Director, Inner Thoughts, Favorability, and Response Candidates.' This stabilizes the AI's reactions and allows for accurate control over the story's progress.
[Series Finale] Tips for Making AI Remember Long Stories and Criteria for Choosing Models Without Failing

Model Characteristics Seen from Measured Data (RTX 4080 Environment)

I have summarized the values when running the models in my actual PC environment. You can see how the balance between speed and quality changes depending on the model size and degree of compression. - gemma4:12b-it-q8_0 VRAM (dedicated memory used for image processing and AI calculations) usage: 13.13GB Speed: 45 tokens/sec (level of reacting instantly) - gemma4:12b (Q4_K_M) VRAM usage: 8.08GB Speed: 64 tokens/sec (extremely fast reaction) In terms of numerical values, the Q4 version is overwhelmingly faster, but caution is required in practical terms. While the Q4 model excels in speed, there was a tendency for the sentence endings to become unstable when continuing long conversations.

  • Speed Benchmark: If it is over 30 tokens/sec, it feels like characters appear instantly.

Criteria for Choosing Models to Avoid Failure

This is a very important point. Models specifically tuned for Japanese roleplay (story interaction) are actually often poor at 'structured output' (answering correctly in a pre-determined format). Therefore, no matter how high the expressive power of the story is, if you choose a model with low adherence to instructions, there is a risk that the system itself will stop working. In this experiment, I prioritized selecting models based on 'whether they can output data as instructed' rather than the skill of creative writing.

  • Cautions for 27B-class models: Running them in Q4_K_M results in about 15.5GiB, which does not fit into 16GB of VRAM. When memory runs out, the calculation process escapes to the CPU, causing the speed to drop to a fraction of the original.

Relationship Between Thinking Mode and Speed

Furthermore, caution is required when turning on the AI's 'Thinking Mode.' When thinking mode is enabled, the AI performs long reasoning (thought process) before providing an answer, so the perceived speed tends to become more than twice as slow. It is important to adjust the balance between speed and accuracy according to your purpose.

  • Speed Benchmark: If it falls below 5 tokens/sec, it reaches the point where the waiting time starts to become noticeable.

Summary: What I Learned Through the Experiment

Through this 'Narratable' series, I was able to reconfirm that to have AI handle complex stories, it is essential to balance 'memory organization' and 'output fixation' on the system side, rather than just using a high-performance model. When running AI on your own PC, please consider the balance between the model's 'expressive power' and 'accuracy to instructions' while keeping memory capacity in mind.

  • Have you ever used AI to write stories or engage in roleplay?
Useful for
  • People who want to try running long stories with AI
  • People who want to know the compatibility between their own PC specs and AI models
  • People who want to know the mechanism to solve the limits of AI memory
Glossary
Context
The range of information that an AI can process and remember at one time.
VRAM
Memory mounted on the graphics board. It significantly affects the AI's operation speed.
JSON Schema
Rule settings to make the AI's answers output in a fixed format such as 'Name: XX, Impression: YY'.
Structured Output
Arranging and outputting the AI's answers into a specific format (such as lists or tables) that is easy for humans or programs to handle.

FAQ

Why does it become slow when running a 27B model on 16GB of VRAM?

According to official information, running a 27B-class model in Q4_K_M results in about 15.5GiB, which does not fit into 16GB of VRAM. When memory is insufficient, the process shifts to the CPU, causing the speed to drop extremely.

Why did it stop working when I switched models?

This is because models for Japanese roleplay may be weak at structured output (accurate adherence to instructions). In the case of a system predicated on a specific output format, it may stop working if you switch to a model that cannot follow those instructions.

Summary

To make AI remember long stories, a mechanism to summarize old statements into a synopsis and output fixation via JSON schema are effective. In model selection, prioritizing adherence to instructions rather than just expressive power is the key to stable system operation.

This article was translated from Japanese by AI; numbers and model names were automatically checked against the original. The original was written with AI from the sources cited there.