The Second Brain Behind My AI Agent

The first time I really felt the problem, I was explaining my own business to ChatGPT for what felt like the hundredth time: my offers, my prices, who my customers are, how I write, and what I'd already decided last month and didn't want to argue with myself about again. Every new chat, I started from nothing, like introducing myself to a very sharp stranger who forgets me the second I close the tab.

Try it yourself. Go back to ChatGPT and ask about something specific you worked through with it three months ago. It won't remember. Not because it isn't clever enough. It is a frontier model, genuinely brilliant. It simply has no memory of you. Close the window and the context is gone.

The idea I borrowed from Andrej Karpathy

The fix didn't come from a smarter model. It came from a way of thinking I picked up from Andrej Karpathy, one of the clearest voices on how these systems actually work.

In his talk Intro to Large Language Models, he describes the model as something like a processor. It does the reasoning, the way a chip does the computing, but on its own it knows nothing about your world. The context window, the small slice of text it can see at any one moment, is its short-term memory, and it is tiny. Everything else, everything it might need to know about you, has to live outside it, in a store the model can reach for when it needs it.

That reframed the whole thing for me. The model is rented, generic, and interchangeable. The memory around it is the part that is actually yours. So I stopped hunting for the perfect AI and started building the perfect memory for the one I already had.

The gap almost everyone is stuck in

Here is the strange part. Almost everyone already has the model. In McKinsey's 2025 research, 88% of companies now use AI in at least one part of the business. Almost nobody is behind on access.

But using it and building on it are two very different things. The same research shows most businesses are still experimenting, using AI like a slightly smarter chatbot, one question at a time. Very few have built it into how the business actually runs. That gap, between the owners who type into a chat box and the ones who own a system, is the whole game. I mapped the stages of it in the four levels of AI adoption. The thing that moves you across it is memory.

What a second brain actually is

I call that memory a second brain. In plain terms, it is a private, growing record of how my business actually works, written down once so I never have to re-explain it. Not a folder of loose files. A maintained, structured store: my offers and prices, my customers, my SOPs, my competitors, the jobs I've done, the way I write, and the decisions I've made and why.

The important word is growing. A chat window resets every morning. A second brain does the opposite. Everything worth keeping goes in and stays in, so the knowledge compounds instead of evaporating. Months in, it holds more about my business than I could keep in my own head, and none of it has been lost.

How I built mine

Mine isn't fancy, and that is the point. It is a set of plain text pages, the kind you could read in any notebook app, kept under version control so every change is tracked and nothing is ever quietly deleted.

It runs on two simple moves. New things land in an append-only inbox the moment they happen: a decision, a result, a lesson, a stray note. Then, every so often, I distill that raw pile into clean, permanent pages, the way you would tidy scattered notes into a wiki. The raw capture makes sure nothing slips through in the moment. The tidy pages are what the agent actually reads. That distilling step is the whole craft.

My agent, I call him Steve, reads that memory with permissioned, read-only access. He can use what is there. He can't quietly rewrite my business behind my back. The memory stays mine. He works from it.

How I actually use it

Once the memory exists, it changes the real shape of my week. A few of the ways I lean on it:

I turn what I watch into knowledge I keep. I get through a lot of YouTube, and most of what you learn there leaks straight back out. So when a video has something worth keeping, I've it transcribed and the real takeaway filed into the second brain. The lesson stops being something I half-remember and becomes something the business can actually use later.

I run a daily journal and a weekly review. At the end of the day I note what got done. At the end of the week, the agent reads the week back to me and helps me see how it actually went: what moved, what stalled, where the time went. It is the review I always meant to do by hand and never did.

I let it read the numbers. This is where AI genuinely shines. It is very good at research and analysis, so I point it at the data across the business, from the website's Google Analytics to social analytics, and it turns rows of numbers into plain-English patterns I can act on.

It lives in my email and my files. It is connected to my Gmail, so it keeps track of what comes in and drafts and sends what needs to go out. It can reach anything in my Google Drive, and any other tool I choose to connect it to.

It handles the moving parts. Scheduling posts, pulling documents, keeping trackers current. Small jobs, constantly, that used to sit on me, right down to designing the odd carousel when a post needs one.

The assistant used to be a person

Step back and look at that list. A few years ago, that was a job. You hired a personal assistant to remind you what was coming up, to review what got done, to send the documents, to chase the email, to keep the calendar honest. Some of it took a whole team.

In 2026, an agent with a second brain does maybe ninety percent of it. Not because it is magic, but because it finally has the two things it was always missing: access to your tools, and a memory of your business. That is the shift. The assistant used to be a person. Now it is a system you own.

Why the memory is the point

None of this works without the memory, and that is the part people skip. The model is the engine: impressive, rented, the same one your competitor can buy. The second brain is the part that is yours, and it is the reason the whole thing gets more useful the longer it runs. That holds no matter how good the engine gets. GPT-6 Astra can finish a whole document on its own and still knows nothing about your prices, your customers, or what you decided last quarter.

Every decision I file, every result I record, every SOP I finally write down becomes context the next answer can stand on. And because nothing is lost, I can rewind. I can ask about a conversation from three months ago, the exact thing a chat window could never hold, and it is there. The business remembers, so I repeat myself less, and every task starts from further along than the last one. It isn't that the software improves itself. It is that your knowledge accumulates, and an agent that can reach all of it quietly does more each month than it did the last.

The one part a competitor cannot copy

An AI Agent with no memory is just a chat window with extra steps. The memory is what makes it worth owning, and it is the one part a competitor can't copy, because it is built out of your business, not theirs.

If you're still working out where your business sits on all this, I mapped it in the four levels of AI adoption. And if you want to see what a second brain for your business would actually hold, that is exactly where the first conversation starts.

FAQ

What is a second brain for an AI agent?

A second brain is a private, growing record of how my business actually works: my offers and prices, my customers, my SOPs, my competitors, the jobs I've done, my writing style, and the decisions I've made and why. It's kept as plain text pages under version control, with an append-only inbox for new notes that gets distilled into clean pages over time.

What is Andrej Karpathy's LLM Wiki?

The LLM Wiki is a gist Andrej Karpathy, a founding member of OpenAI, posted on 4 April 2026. It describes a two-layer memory setup: raw sources the agent reads but doesn't edit, and a wiki folder the agent writes and owns entirely, holding summaries, entity pages, and an index of everything in it.

How long does it take to set up a second brain?

That's about twenty minutes of setup: make a folder, open Claude inside it, and create two subfolders, one for what you dump in and one for what the agent writes back. Filling it with enough of your business to be genuinely useful takes months.

How much of the assistant's job can an AI agent with a second brain handle?

About ninety percent of what used to be a personal assistant's job: reminding you what's coming up, reviewing what got done, sending the documents, chasing the email, and keeping the calendar honest. It can do that much because it's finally got two things it lacked before: access to your tools and a memory of your business.

Update, September 2026: the document itself, and the two-folder setup

People kept asking for the actual document behind this, so here it is: the same inbox-and-wiki shape I described above, except this time it's the real paper Andrej Karpathy wrote, and the exact folders he tells you to build around it.

The document is a gist Karpathy posted on 4 April 2026, called the LLM Wiki. Karpathy is a founding member of OpenAI, and the gist itself is a short idea file with no app to install, meant to be pasted straight into an agent like Claude Code. It tells the agent to run two layers at once. Your raw sources stay untouched, the agent reads them but never edits them. The wiki is the folder it writes and owns entirely, summaries, entity pages, comparisons, an index of everything in it. Feed it a new source and it reads the source, talks the key points through with you, writes a summary page, and updates the index. Ask it a question and it searches the wiki, reads the pages that matter, and answers with a citation back to where it found each fact. And when an answer is good enough to keep, it gets filed back into the wiki as its own page, so it doesn't just sit in your chat history and disappear.

The setup goes like this, in the order I actually build it. First, make a folder and open Claude inside it. Then inside that, make two more: one for what you dump in, and one for what Claude writes back. Dump in whatever you already have sitting around, last year's proposal, the price list you keep retyping. Then ask it something you would normally have to go dig for yourself. When I asked what I quoted a client for a rebuild, it answered from my own files and named the exact file it pulled the number from.

Every answer it gives gets saved, so it gets smarter.

One catch, and the reel only had a few seconds to say it. Setup is twenty minutes. Filling it takes months.