My LLM note-merge test passed, but the fixed code never ran once

The Obsidian plugin I’m building has an LLM read source material and organize it into notes, one per concept, and when two notes on the same subject appear, it merges them into one. In Part 7, I changed the criterion for that merge judgment to “if two notes cover the same subject at the same scope, merge them, even if they come from different sources.” Thanks to that, the English note “Claude Code” and the Korean note “클로드 코드” (Claude Code written in Korean) merged for the first time. This post is about what happened while fixing and checking two small problems left over after that. To give away the ending: the real test I ran after the fixes passed cleanly, but the fixed code never ran once in that test.

The merged note’s name didn’t match its filename (frontmatter title)

When a Korean note and an English note are merged, I had set the result to be written in Korean. But when I opened the merged note, the filename was Claude Code.md while the title inside the note (the title in its frontmatter) was “클로드 코드.” The result of merging three “psychological safety” notes was similar: the filename was 심리적 안전감.md (“psychological safety”), but the title was one the LLM had newly made up, “The Core Principles of Psychological Safety and Leadership Strategy.”

It wasn’t just awkward to look at. This plugin finds duplicates by grouping notes with the same title, and when it does, it looks at the title inside the note before the filename. If the title differs from the filename, the note can’t be grouped with a same-named note in the next scan. I made the plugin set the merge result’s title to the filename right before saving. The plugin keeps filenames from colliding across the whole wiki, so the titles are just as unique.

Once the names matched, one more thing stood out. The body was Korean, but the filename was English. So when merging a note with a Korean name and a note with an English name, I made the plugin pick the Korean-named one as the merge target. Now the two notes merge into 클로드 코드.md.

Changing the target made “클로드 코드” show up in English notes (wikilink alias)

When I tried this fix on real notes, they merged into 클로드 코드.md as intended. But this time other notes went wrong. Four notes written in English had links that used to point to Claude Code, and after the merge those links showed up in the sentence as “클로드 코드.” Korean had wedged itself into the middle of English sentences.

The cause was the code that moves links to the new note after a merge. If a link’s alias (the visible text) is the same as the old note’s name, this code changes it to the new note’s name. That behavior is intentional. When notes are split or merged and the target changes, the old name no longer describes the new target properly. But “Claude Code” and “클로드 코드” are names for the same thing in different languages, so the old name was still a correct name. Links that had only a path and no alias had the same problem, since they display the new filename as is.

So when only one of the old and new names contains Korean, that is, when only the language changes, I made the plugin keep the text that was visible before. If the alias is the old name, it stays; links that had no alias get the old name added as an alias. The link becomes [[…/클로드 코드|Claude Code]], and in English notes it still reads “Claude Code.” Between names in the same language, the original behavior stays. I added unit tests and shipped a new version.

I reran it on real notes, and it came out clean (LLM nondeterminism)

I installed the new version in the test vault and ran it the way I always do. I feed in six source documents from scratch to generate notes, then repeat the merge/split suggestion scan until it produces no suggestions twice in a row. It stopped after two scans. There were no broken links, every note’s title matched its filename, and nowhere did “클로드 코드” show up in an English note.

Before writing it down as a pass, I opened the merge log. This time, the Korean source had also created its note under the English name “Claude Code” rather than “클로드 코드.” Two notes with the same name appeared, so they simply merged, and since the name didn’t change, the links kept the old name. The scene where a Korean-named note meets an English-named note never happened at all. The two paths I fixed, the code that picks the Korean name as the target and the code that preserves link text, never ran once in this test. “Zero places where 클로드 코드 shows up in an English note” was a number that came not from the fix, but from there being no chance for it to happen.

As long as an LLM is writing the notes, the same source doesn’t produce the same scene every time. The note that was named “클로드 코드” last time was named “Claude Code” this time. Run it again, and there’s no telling what comes out.

I brought back the exact scene right before the merge (restoring test state from a git commit)

So instead of leaving it to luck, I built that scene myself. This test records the vault state in a git repository every round, and from the previous test there was a commit from right before the merge: English Claude Code.md and Korean 클로드 코드.md sitting side by side, with four English notes pointing to Claude Code. I restored that moment’s 41 notes, 6 source documents, the work log, and even the cache used to find duplicate candidates, 50 files in total, exactly as they were. I restored the cache too so that the same candidates would be picked up and the “same subject” verdict made back then would be reused as is.

When I ran one scan from that state, it suggested merging Claude Code into 클로드 코드. After running it, I compared the six links in the English notes one by one against how they were before the merge.

All six pointed to the new note while reading “Claude Code.” The merged 클로드 코드.md had a title matching its filename, a Korean body, and the facts from both notes intact. Only now did I confirm that the fixed code had actually run.

Bonus: I almost believed an “in progress” report (AI agent progress reports)

I had originally handed this test to another AI agent. On the first try, the agent replied “the test is running as a separate task” and ended there. When I opened the repository where test results are recorded, there wasn’t a single new record, and the vault still had the old version of the plugin installed. In reality, it hadn’t even started. When I asked again, it stopped partway this time, and again nothing changed. Only after I handed the job from scratch to a new agent that didn’t inherit the whole conversation history did the test run to the end. Just as with judging the test results, progress also had to be checked through the records left behind, not through the report.

Closing thoughts

A test passing and the fixed code running are two different statements. With ordinary code, the same input goes down the same path, so the two mean nearly the same thing, but once an LLM is in the middle, the same input can produce a different scene. Judged by its results alone, this test was flawless, yet the very scene I wanted to check wasn’t in it.

When confirming a pass, I should have looked first at “did it actually go down the fixed path?” before the result numbers. If it didn’t, I can restore the state from the moment that scene existed and run it again. Recording the state every test round is what let me pull out that exact moment this time.