The Obsidian plugin I’m building has an LLM read source material and organize it into notes, one per concept. As sources pile up, two notes on the same subject sometimes appear, so the plugin shows both notes to an LLM and asks: “Same concept, so merge them? Or different concepts, so keep them apart?” This post is about replacing the rule behind that merge judgment entirely.
Part 3 ended like this. A pair of notes on the AI labor market had been written into the rule as an example of “a case that must not be merged.” When I actually merged them, there was no reason to block it, so I deleted that example. But what I deleted was one example. The rule that produced it, the principle that “if unrelated sources merely explain the same general topic, they are different concepts,” stayed in place. The remaining example, a “psychological safety” pair, stayed too, as a counterexample that had never been shaken.
Notes on the same product didn’t merge — Claude Code and 클로드 코드
The problem surfaced in a different experiment. While measuring whether this merge judgment could be handed to a decision-only model (the decision-model series), I fed in the plugin’s judgment prompt as is, and it answered “different concepts” all three times for a “Claude Code” note and a “클로드 코드” note (the Korean name of Claude Code). Same product. When I got the reasoning back, it said that while both covered the same product, one was about Uber’s adoption and the other about personal use, so the specific events didn’t overlap and they were separate concepts.
The model followed the rule exactly. The rule used “do the sources point to the same specific event” as its test, and since that test had no stated scope, it was applied to product and company notes too. My first instinct was to add a sentence saying “unique products and companies are an exception.” But before fixing it, I decided to go back and look at where this rule came from in the first place.
Tracing the rule back, nobody had ever actually merged them (LLM merge criteria)
Going back through the change history, it started with a single judgment. Two pairs of notes whose titles matched across English and Korean (“Psychological Safety”/“심리적 안전감”, “Claude Code”/“클로드 코드”) came up as duplicate candidates, and when I opened the bodies, each note in a pair came from a different source article. Seeing that, I called them “false positives that shouldn’t be merged because the source articles differ,” and the next day that judgment became a rule.
One step was missing. “The source articles differ” was a checked fact, but “so merging them would make things worse” was a guess nobody had checked. The only recorded test of that guess was the AI labor market pair from Part 3, and that test came out the opposite way from the rule. For the psychological safety pair that remained as the rule’s representative example, I couldn’t find any record of actually merging it and looking at the result either.
There was one more mismatch. The plugin has a separate step that judges two notes whose titles collide when notes are created, and that step’s prompt gave an example saying to merge notes that each independently re-explain the same technical term. Given the same two notes, the note-creation step would merge them, while the step that later scans the wiki would say to keep them apart.
Karpathy’s LLM Wiki went the other way (Karpathy LLM Wiki)
I reread Karpathy’s LLM Wiki, the concept this plugin is based on. It says that when a new source arrives, the LLM “updates relevant entity and concept pages across the wiki.” Rather than creating a separate page, it updates the concept page that already exists. When sources disagree, it describes “noting where new data contradicts old claims, strengthening or challenging the evolving synthesis,” so instead of splitting, the differences are marked and synthesized within one page. The content specific to each source is handled by a summary page kept per source.
If a book and a manager’s guide each explained psychological safety, Karpathy’s approach would put both sources’ perspectives together on a single “psychological safety” page. My rule was blocking exactly that.
This time I actually merged them instead of arguing — duplicate note merge test
Instead of reaching a conclusion by prediction, I did for the remaining pairs what I’d done for the AI labor market pair in Part 3. From my test vault I picked five pairs where “unrelated sources explained the same general concept,” merged each three times with the plugin’s real merge prompt, compared the merged result directly against both original notes, and then ran a scan on the merged note to see whether it suggested splitting it again.
| Pair | Original facts preserved | Suggestion to split again |
|---|---|---|
| Psychological safety (a manager’s guide / an article on leadership strategy) | Preserved all 3 times | None (checked without surrounding notes) |
| Korean National Pension lump-sum refund (different pension articles) | Preserved all 3 times | None |
| Model-based design (an autonomous-driving article / a concept explainer) | Preserved all 3 times | None |
| Digital twin (a robot-learning article / a climate-model interview) | Preserved all 3 times | None |
| AI Adoption (AI diffusion statistics / how organizations adopt AI) | Facts preserved | 2 out of 3 times |
Four pairs merged without any problem. The pension pair even produced something good. The two articles described the refund’s interest rate, and whether money received could later be paid back, differently, and in two of three runs the merged note pointed out that “the sources describe this differently.” Kept apart, nobody would have noticed that difference. In effect, the “noting where new data contradicts” that Karpathy describes happened with the current merge prompt.
AI Adoption was different. The name was the same, but one side covered how widely AI had spread through society and the other how organizations bring AI in. All the facts survived, but the merged note became too broad, and when I scanned again, it suggested splitting it into “organizational adoption” and “workforce impact.” That was the signal that a pair with the same name but a different scope shouldn’t be merged.
The new criterion: same subject at the same scope (LLM prompt fix)
So I changed the rule to this. If two notes cover the same subject at the same scope, they are the same concept even if their sources are unrelated and their examples differ. These, however, are different concepts: notes with the same name but a different central topic, one note being a sub-concept of the other, different things that merely share a name (people with the same name, different terms that share an acronym), and a concept versus a named methodology for practicing it. Whether the sources are the same or different is not, by itself, grounds for the judgment. I kept the principle that when in doubt, keep them apart. Products, companies, and people also count as “the same subject,” so the Claude Code problem is solved without an exception clause.
The first wording was too strict. Two notes from different articles naturally emphasize different things even when they cover the same concept, and the model read that difference as “a different scope.” It called many of the four pairs that had merged without problems in the table above “different concepts,” getting only 7 of 15 judgments on the five pairs right. After adding “explaining the same concept with different examples or emphasis is not a difference in scope” and “a sub-concept is a separate concept with its own name,” it got all 15 right. On a separate set of 40 note pairs with predetermined answers, the old rule got 96 of 120 right and the new rule 119. Pairs in a sub-concept relationship stayed “different concepts” even after the wording was loosened.
I applied this rule in the plugin so that the note-creation step and the wiki-scanning step both use it. The two steps no longer give different answers about the same pair of notes.
Running it again on real notes, the old judgment was left in the last gate (LLM note merge verification)
Since I had tuned the wording on five pairs, I checked again with real notes. I emptied the test vault and added six new sources: an English and a Korean article on Claude Code, two different pieces on psychological safety, and two English pieces on AI adoption. After creating the notes, I ran the merge/split suggestion scan, executed any suggestions, and repeated until two scans in a row produced no suggestions at all. It stopped after three scans.
Claude Code and 클로드 코드 got a merge suggestion in the first scan, and executing it produced a single note containing the facts from both. But merging an English note with a Korean one left English and Korean joined inside a single summary sentence, and the language changed from section to section. The merge prompt only said “follow the source language,” and never said which one to use when there were two. I added a sentence saying to write the whole note in Korean when merging notes in different languages, merged the same pair three more times, and all three came out in Korean. Merging two English notes stayed in English.
Psychological safety was blocked. The third scan suggested merging the two notes, but an audit step that checks once more right before execution rejected it. The reason was this (translated from Korean):
Document 1 covers the higher-level concept of psychological safety and document 2 covers the sub-topic of leadership strategies for promoting it, so this is a containment relationship.
This audit doesn’t receive the note bodies; it only sees titles, tags, and tables of contents. So it read the title “Leadership strategies for promoting psychological safety” as a sub-topic. In reality that note was the broader one, covering psychological safety starting from its definition. When I changed the rule, I’d left this audit alone, figuring it only checked “don’t merge if it’s a sub-concept” and wouldn’t conflict with the new rule. But the old reading was still there in exactly how it judged sub-concepts.
I added two sentences to the audit: a note that explains a concept along with ways to strengthen it is the same concept, and a title with a word like “method” or “strategy” attached isn’t, by itself, a reason to treat it as a sub-topic. At first I also included “an explanation applying the same concept to a particular situation isn’t a sub-concept either,” but in an audit that sees only titles and tables of contents without bodies, that let genuine sub-concept pairs through too (two of five times, and once). After removing that sentence, the pair that had actually been rejected passed all five times, and the four sub-concept pairs the original audit used to reject were still rejected all five times.
The AI adoption side I couldn’t confirm this time. The two pieces never produced notes with the same name, so a pair with the same name but a different scope never appeared.
How far can these results be trusted
- The wording was tuned while looking at five pairs. 15 out of 15 may be a number fitted to those five pairs. I checked once more with real notes, but that was six sources, one round. Whether it keeps pairs with the same name but a different scope apart hasn’t been confirmed with real notes yet.
- More merging exposes more of merging’s weaknesses. This plugin still has a problem where content that existed on only one side occasionally drops out when merging two notes from different sources. It didn’t show up in these 15 merges, but in one of three merges of notes in different languages, one body sentence was left only as a related-page link. It’s a problem I’ve seen about once in four times in real use, so I need to keep watching it.
- Merged notes are hard to undo. As Part 3 showed, once notes are merged, the next scan doesn’t split them again. A wrong merge has to be untangled by hand, so the “when in doubt, keep them apart” principle stays.
Final thoughts
What Part 3 taught me was “after fixing a criterion, recheck the original case that made you create it.” This time it was one level up. Once I’d confirmed the original case was wrong, I should have rechecked the rule that came from it too. Deleting one example and keeping the principle was like deleting the evidence for a wrong conclusion while keeping the conclusion. The remaining example being “a counterexample that had never been shaken” was also only because nobody had ever actually shaken it.
What overturned the rule wasn’t making the judgment more sophisticated; it was actually carrying out the judgment. “Merging will make things worse” was a claim you could check by merging, and checking it cost a few minutes and a few cents.
And I nearly made the same mistake once more. While changing the rule in three places, I left the fourth, the pre-merge audit, alone because “it judges something else, so it doesn’t matter.” That was a guess too, not something I’d checked, and only running it on real notes revealed the old reading left there. Once you change a criterion, you have to actually run not only the places that use that criterion but also the places that make similar judgments.