Pushing the Limits of 16GB VRAM! Techniques for "Memory Sharing" to Coexist Text AI and Image Generation

For those who think, "I want to run AI on my own PC, but I don't have enough memory, so making images makes the text response slow..." we are here for you. This time, we deliver practical insights on how to efficiently run multiple AIs within the limited resources of 16GB VRAM.

The Identity of the 16GB VRAM "Wall"

When running AI on your own PC, one of the most important specs is VRAM (video memory). This is the memory mounted on the graphics board, serving as a kind of "workbench" for running AI models.

In this experiment, I used the relatively high-performance RTX 4080 (16GB), but I hit a wall. Loading the model for text generation (gemma4:12b-it-q8_0) consumes about 13.1GB of memory. Meanwhile, since the image generation software Forge also requires memory, it is difficult to keep both "loaded" at the same time.

• Consumption by text AI (gemma4:12b-it-q8_0): Approx. 13.1GB
• Conflict with image generation software (Forge): Memory runs out if loaded simultaneously

Impact on Speed: "Comfort" Changes Just by Keeping Software Resident

The perceived speed when running AI is heavily influenced by the VRAM status. As a result of the experiment, significant changes in text AI generation speed were observed just by having the image generation software (Forge) running.

Specifically, what was at a level that outputs in "instant" (62 tokens/sec) when nothing was launched dropped to less than half—33 tokens/sec—just by launching Forge. Furthermore, if you load the text generation model on top of that, it drops down to 14–18 tokens/sec.

These numbers are close to the perception of "short wait but usable." I learned that clever tricks are needed to avoid keeping everything resident to maintain comfortable speeds.

• Forge not launched: 62 tokens/sec (outputs instantly)
• Forge launched: 33 tokens/sec (comfortable, faster than reading)
• Model also deployed simultaneously: 14–18 tokens/sec (short wait but usable)

Solution: Implementing "Dynamic Memory Management"

To solve this problem, I introduced a mechanism on this account where they "take turns yielding space only when necessary." Basically, Forge is closed when not performing image generation and is launched only at the exact moment images are needed.

However, that alone was insufficient. Even if the text AI is removed from VRAM right before image generation, a problem occurred where the system's background "post-processing" would automatically reload the model immediately after it was removed.

Therefore, I introduced a mechanism to control the order of processing. By adding mutual exclusion (determining the order so multiple processes do not overlap)—specifically "stop post-processing during image generation and make image generation wait during post-processing"—I succeeded in avoiding VRAM conflicts.

• Step 1: Release the text AI from VRAM just before image generation
• Step 2: Temporarily pause background post-processing during image generation
• Step 3: Make image generation wait until post-processing is finished

Tips to Increase Efficiency Further

Finally, I will introduce operational tricks for smoother workflow. For example, when you want to create icons for multiple characters at once.

If you generate images one by one, software launching and memory swapping occur every time, causing wait times to accumulate. To prevent this, I use a method where I "create all the prompts (instructions for the AI) in advance and output them all together in a single image generation process."

With this trick alone, it is possible to efficiently gather materials while minimizing wait times.

• Do not generate individually when creating multiple icons
• Prepare tags (instruction content) for everyone first, then generate in bulk
• This minimizes the number of software launches/closures and memory swaps

Summary and Next Time Preview

In an environment with 16GB VRAM, it is possible to balance both text generation and image generation by creating a mechanism where they "share memory seats" depending on the situation, rather than keeping all AIs running at all times.

In the next installment, Part 5, I will explain more advanced content: "a mechanism for remembering long stories" and "criteria for model selection" based on your goals.

• Next theme: Mechanism for remembering long stories
• Next theme: Criteria for model selection

Question to the Readers

Do you ever feel a shortage of memory when running AI on your PC? If so, please let me know what kind of situations you are struggling with.