I write most posts on this blog together with a coding agent (Claude Code). The agent writes a draft from the records I left while working, and right before publishing, the agent checks the draft’s facts again. It checks against three sources: notes summarized along the way during the work (the agent’s long-term memory files), the repository’s commit history, and the raw conversation stored in Honcho (Korean only). Honcho is a memory server that stores my conversations with the agent in full.
This post is about something that happened while publishing a draft I had written a month earlier. The agent corrected the draft using only its summary notes, and one of those corrections turned an anecdote that had been right into a wrong one.
The month-old draft and the summary notes disagreed (AI agent long-term memory)
The draft was the story of testing a plugin by remote-controlling Obsidian on another PC. I had written it in mid-September, so before publishing I had the agent check it against the records from that time. The agent opened the notes it had summarized from that work and found several places where they differed from the draft. The draft called the test PC a spare PC in another room, but the notes said it was an old Linux PC at home. The draft and the notes also described a bug in the transfer program differently.
The button anecdote differed too. The draft had a story that “the last of several identical-looking buttons was pressed, and it turned out to be one card’s individual run button, not the batch run.” The notes didn’t have that story. Instead they recorded a different one: “the batch run button under a card quietly does nothing if nothing is checked.” Using the notes as its evidence, the agent replaced the draft’s button anecdote with the one from the notes.
The raw conversation had both (Honcho conversation logs)
While the agent was making those changes, I told it, “Check the Honcho memory too.” When the agent looked up the raw conversation from that period in Honcho by date and read it, both button anecdotes had actually happened. On the same day, the individual run button was pressed by mistake in the morning, and that night the batch run button did nothing. The notes kept only the second anecdote and the tip learned from it; the first one had been dropped in the summarizing. The draft was right, and the change made by trusting the notes was wrong.
The raw conversation corrected other things too. What started this work wasn’t, as the draft said, that testing by hand in front of the screen every time was tedious. Until then I had to publish a release every time I tested the plugin, and I wanted to change that order. The reason for changing how files were transferred was right there in the raw conversation as well. The server I was working on had no key to connect to the home PC, so the existing method was blocked, and the starting point was me asking, “Can it be done without using scp?” The file size also differed from the number in the draft. The draft, the notes, and the raw conversation disagreed with one another, and of the three, only the raw conversation was what had actually been said at the time.
Summaries merge and drop anecdotes (limits of LLM memory summaries)
Summary notes are very useful. They briefly hold tips that keep the next task from repeating a mistake, the reasons behind decisions, and traps to avoid. The problem is that brevity. Notes are cut down to fit “what to do next time,” so when something similar happens twice they merge it into one, and they drop an anecdote whose lesson overlaps with another. This time, too, the notes were accurate as a tip: “find buttons by their text, and press select all first.” But for asking “what happened back then,” something was missing.
A blog post records exactly that, “what happened back then.” What matters is not the tip but the order of events, the numbers, and what triggered them. If you use a summary as evidence when checking those, what the summary dropped becomes “something that never happened.”
I added a raw-conversation check to the publishing process (AI content fact-checking)
After that, I added one more step to the publishing process. If a post describes past work, it gets checked not only against the notes and commit history but also against the raw conversation from that time. Honcho also has a feature that answers by summarizing conversations, but I don’t use it for fact-checking. That would mean using another summary to check a summary. I find the raw messages by date and read what comes before and after them directly. And if they disagree with the draft or the notes, the raw conversation wins.
Closing thoughts
The agent I’d trusted with fact-checking didn’t get it wrong out of laziness. On the contrary, it found evidence and made corrections. It’s just that the evidence was a summary, and a summary doesn’t tell you what it left out. It read an anecdote missing from the notes as “something that never happened.”
The side doing the checking needed the same question when choosing its evidence: is this the record from that time, or a later, shortened version of that record? This time, one line, “Check the Honcho memory too,” closed that gap, and now that line is built into the process.