[My Homegrown AI Diary Vol. 3] The Process and Mechanism of Surpassing 10,000 Knowledge Base Entries in Just 38 Days
Hello, this is Shachi Lab. This time, I am sharing the progress of my ongoing "Project to Raise My Own Personal AI." In just a short period of 38 days since starting the activity, the number of knowledge entries the AI references has exceeded 10,000. I will explain the behind-the-scenes details of how this knowledge is accumulated and utilized through specific mechanisms.
- Knowledge Accumulation Status: Integrating diverse data centered on headlines of trending AI articles
- Mechanism Explanation: Managing data in a "searchable by meaning" format using RAG (Retrieval-Augmented Generation)
- Resource Optimization: Efficient processing utilizing the CPU and reserving the GPU for text generation
- Latest Trends: Keywords currently gaining attention as seen from the accumulated data
![[My Homegrown AI Diary Vol. 3] The Process and Mechanism of Surpassing 10,000 Knowledge Base Entries in Just 38 Days](/img/labnote-3-20261110-005023.png)
Current Status of the Knowledge Base
On the 38th day of the project, the total number of accumulated knowledge entries reached 10,312. Since there were 5,580 entries at the last update (Vol. 2), this means 4,732 pieces of knowledge were added in this period alone.
The breakdown of the currently accumulated data is as follows:
- Trending Headlines: 10,125 entries (extracted from trending AI articles on note, Zenn, Qiita, and Hatena Bookmark)
- Past posts from this account: 103 entries
- Articles from this blog: 61 entries
- Measured data from this PC: 23 entries
By combining external trend information with my own past activity data in this way, I am building a multi-faceted knowledge base.
- Handling of External Information: Regarding others' articles, considering copyright and information accuracy, I do not save the full text and only accumulate the "headline," "short summary," and "link."
The Mechanism to Make Knowledge "Searchable by Meaning"
The core of this project is not simply accumulating data, but making it so that the AI can search for content by "meaning." This mechanism is called RAG (Retrieval-Augmented Generation). RAG is a system where, when an AI generates an answer, it finds relevant information from pre-prepared external knowledge and constructs an answer based on that.
In this system, I have adopted "bge-m3" as the "embedding model" for searching by meaning. This makes it possible to extract information based on context and semantic proximity, rather than just keyword matches.
In actual operation, when writing a blog post, the system automatically retrieves up to 6 pieces of knowledge related to that theme and presents them to the AI. This allows the AI to generate more accurate answers based on my past data.
- Processing Optimization: The knowledge search (embedding) process is executed using only the PC's CPU. This reduces the load on the GPU (a processing unit specializing in image processing and high-level calculations) required for AI text generation, achieving efficient operation.
Latest Trends Visible from the Data
I extracted keywords that were particularly frequently discussed over the past two weeks from the accumulated data. Currently, the following terms are gaining attention in the field of AI:
- Claude (61 entries)
- Claude Code (57 entries)
- Agent (55 entries)
- ChatGPT (50 entries)
- Local LLM (40 entries)
(*Local LLM: Large language models operated in a local environment such as your own PC)
When I actually retrieved knowledge using the keyword "Claude," headlines regarding how to use it at work and specific operational details appeared at the top. It confirms that as knowledge increases, I am able to accurately extract information that is closer to my intended meaning.
- Future Outlook: I will explain in detail in the next report (Vol. 4) how the accuracy and diversity of the information the AI retrieves will change as more knowledge piles up.
- Those interested in building their own AI
- Those who want to understand the mechanism of RAG with concrete examples
- Those who want to check the latest AI trends with data
- RAG (Retrieval-Augmented Generation)
- A mechanism where an AI searches for relevant information from reliable external knowledge sources when generating an answer and constructs the response based on that information.
- Embedding Model
- A model that converts data such as sentences into a series of numbers (vectors) so that a computer can calculate "semantic proximity."
- GPU
- A processing unit specializing in image processing and large-scale parallel calculations. High performance is required for AI training and inference.
- Local LLM
- A large language model operated in a local environment such as your own PC without going through a cloud service.
FAQ
Why don't you save the full text?
Because they are others' articles, I have a policy of only accumulating the headline, summary, and link out of consideration for copyright.
Why are you running it on the CPU only?
To concentrate the GPU on the AI's text generation processing, I have configured the system to perform the search process (embedding) efficiently on the CPU.
In this article, I delivered Vol. 3 of "My Homegrown AI Diary," where I expanded the knowledge base to over 10,000 entries in just 38 days. Using a mechanism called RAG, I accumulate AI trend articles gathered daily as headlines and summaries. I also explain in detail the process of training an AI on my own personal knowledge through an efficient system configuration utilizing the CPU.