I Built Self-Evolving Claude Code Memory w/ Karpathy's LLM Knowledge Bases
📚 Main topics
- Karpathy’s knowledge-base architectureThe video walks through a system that ingests raw external information, uses an LLM to compile it into linked wiki pages, and lets agents query that organized knowledge. Obsidian provides a way to browse the documents and their connections. 0:30
- A compiler-inspired workflowRaw articles and notes are processed into a searchable wiki, while integrity checks look for gaps, stale information, and broken links. An index file helps agents navigate the knowledge without requiring a vector database or elaborate RAG setup. 2:34
- Memory for coding projectsThe creator adapts this architecture to internal project information, capturing Claude Code conversations as daily logs and using the Claude Agent SDK to extract structured, cross-referenced knowledge. 8:45
- Hooks and memory processingSession-start hooks load the system instructions and knowledge index; session-end and pre-compaction hooks summarize conversations into daily logs. A daily flush process promotes log content into the wiki. 12:25
- Sponsor segmentInsForge is presented as a platform that combines backend services, model routing, and hosting, managed through a CLI and agent skill. 9:17
✨ Key takeaways
- A browsable index can guide searchAn actively maintained index gives the agent a starting point for navigating Markdown files and following links between related ideas, avoiding the need for semantic search in this setup. 5:07
- Project memory captures decisions and lessonsThe system preserves takeaways from coding-agent conversations so the agent can answer project-specific questions without repeatedly analyzing the whole codebase or relying only on Git history. 13:58
- The knowledge base compounds over timeNew conversations add to the daily logs, and the flush process can promote useful ideas into the wiki, making future answers more informed. 16:35
- The setup is customizableThe prompts and scripts are accessible, and the coding agent can use its understanding of the system to help modify it. 16:04
🧠 Lessons learned
- Keep the architecture simple and inspectableSeparate raw logs from curated knowledge, explain the system to the agent, and make the index easy to maintain. 7:11
- Use hooks to automate captureSession-end and pre-compaction events provide natural points to summarize conversations before their context is lost. 14:29
- Treat memory as an evolving systemAnswers and research from later sessions can become additional knowledge, improving the context available to future sessions. 17:07
🏁 Conclusion/next steps
- Try the setup in a codebaseThe creator recommends using the provided prompt in Claude Code to install the system, and suggests opening the project as an Obsidian vault to browse its logs and wiki. 10:51
- Explore further resourcesThe video points viewers to the Dynamis community workshop for building a second brain with Claude Code and the Claude Agent SDK, and asks viewers to like and subscribe. 18:39
Transcript excerpt
0:00 Let's face it, every week you and I are playing the game, what's the latest and greatest in the AI space, right? It changes every single week. And right now, everyone is focused on LLM knowledge bases, which originated from this tweet from Andre Karpathy. And there are a lot of really cool ideas here that I want to get into with you. And I built my own memory system on top of this. I think you're really going to like it's simple, but super effective. So, more on that in a bit, but let's get into a bit of context here. So Karpathy starts by saying something I'm finding
0:30 very useful recently is using LLM to build personal knowledge bases for various topics of research interest. So taking external information bringing it into our own system and organizing it in the best way for agents to query. And this is a big use case for AI second brains right now which is something I've been focusing on a lot recently. And I love seeing that he's using Obsidian as a core part of his stack. I always call it my canvas for working with things with my second brain. So cool to see that. And he really gives us his entire
🔒 The full, searchable transcript is available with Pro.
Related summaries
Google is Quietly Revolutionizing AI Agents (This is HUGE)
Code 100x Faster with AI, Here's How (No Hype, FULL Process)
La fin de la vulgarisation.
Les effets indésirables de la psychothérapie
Former Intel CEO: Why This is the Best Time to Build Hardware
