When the Obsidian plugin I’m building uses an LLM to split one long article into several notes, the notes that come from the same article (I call them “sibling notes” here) tend to repeat the same facts to each other. To catch that duplication, I had built a filter: turn each paragraph into an embedding vector, keep only the pairs that look similar, and ask the LLM about each of those pairs whether it is a real duplicate.
Part 5 ended on this point: embeddings are a great filter for narrowing down candidates, but no matter how good the model, some cases still slip through. A few days after writing that sentence, I ran straight into that limit again.
Same leftover cases, 2 runs out of 3 — embedding similarity misses paraphrases
While verifying a bug in the feature that splits notes (split), I had run the same article through note creation (ingest) three times in a row. When I later reopened the records I happened to have saved from those three rounds, the “cases the filter can’t catch” from Part 5 had come back in 2 of the 3. In one round, a sentence saying a chair company had halved its customization options appeared almost word for word in two notes. In another, the same success story was repeated in different wording. This wasn’t a one-off regression; it was a real limit that kept showing up.
Breaking paragraphs down further and computing similarity per sentence didn’t help. Even at the sentence level those two leftover cases stayed at 0.62 to 0.66, a range no threshold could pull into the candidate list. The problem wasn’t the grade of the model but the structure of the approach. Duplication that has been paraphrased, or diluted by the sentences around it, never comes out as “close” by mathematical similarity in the first place.
Not filtering candidates, handing over everything — finding duplicate paragraphs with an LLM
So I decided to change the structure entirely. Until then it was two stages: turn paragraphs into vectors to narrow down candidates, then ask the LLM about each candidate pair separately. Instead, I switched to a single call: give the LLM every sibling note from one note-creation run at once and say, “find the places in here that state the same fact twice, fix them on the spot, and give them back.” With no candidates narrowed down in advance, paraphrase or dilution, it’s the eye reading all the notes that makes the call.
I tested this on 5 groups of notes I had actually created before (13 to 19 notes each, mixed Korean and English, with domains ranging across engineering, leadership, psychology, and robotics).
It flagged the hub note as a false positive — false positives in LLM duplicate detection
The first results were disappointing. In a group where one of the sibling notes also served as an index or hub that introduces the others, precision dropped to around 55%. The cause was simple: it’s a normal design for a hub note to summarize its subtopics in a sentence or two up front, and this approach flagged even that as “overlapping with other notes.” In one 15-note group, without this rule, only about 6 of 11 flagged cases were real violations.
One exception rule changed the precision — reducing false positives with a prompt
I added one rule to the prompt stating that a hub or index note briefly summarizing its subtopics is not duplication. It had three conditions: the mention is short, it doesn’t repeat the subnote’s specific numbers, quotes, or story, and it comes with a link or pointer to the subnote. Only when all three hold is it accepted as normal; if even one is broken (for example, it looks short but copies the exact same number), it is still flagged as duplication.
The results changed. The hub group that had been around 6/11 flipped to 5/5 once the rule was in. The two flat groups (10/10, 8/8) and the group where one person’s note summarizes the whole talk (essentially all of 17) kept high precision, and the last group, with 19 notes, got better at correctly leaving out ambiguous short summaries.
What the old approach recorded as “clean” wasn’t
One side effect came up that I didn’t expect. The new approach found violations even in two groups (13 and 15 notes) that I had recorded as “clean” under the old two-stage approach. A manual spot check simply couldn’t compare every pair among 13 to 15 notes; those groups had never actually been clean. What the candidate-narrowing filter had missed wasn’t just a couple of leftover cases. It was a whole area I had believed was “fine” without ever properly verifying it.
Ripping out the embedding infrastructure — down to the text-embedding-3-large wiring
Once the precision was confirmed, I changed the code. Paragraph extraction,
batched embedding calls, cosine similarity, and the verification call for each
candidate pair: I removed the entire embedding candidate filter pipeline that
Part 5 described in detail and replaced it with the single call above that hands
over all the notes at once. The text-embedding-3-large wiring that had been
used only for this pipeline (the very model switch in Part 5 that flipped the
ranking) no longer had anything to do, so it went too.
In terms of LLM calls, each note-creation run went from “1 generation + K verifications” (K sometimes went over 20 for groups with many siblings) to a fixed “1 generation + 1 review.” An approach that doesn’t filter candidates ended up making fewer calls.
Running it for real, seven times
In the development Obsidian window I run myself, I put the real source article of each of the five groups I had tested back in, one at a time, and ran note creation (ingest). The two tricky groups (hub structure) twice each, the rest once each: seven real runs. Each time I didn’t just look at the log and move on; I opened the notes that were actually written and compared them. Here is how one round checked out: the note that held the step-by-step description stayed intact, and the sibling note that had repeated it was compressed into exactly one sentence, “details are covered in [[that note]].” All seven runs gave the same result.
Takeaways
I assumed that removing the embedding filter would cost either speed or precision. In practice both got better: fewer calls, and violations that the filter had kept from ever becoming candidates were now caught. The very idea of a filter rests on the premise that it’s “a trade-off to save on expensive judgment,” but that trade-off had actually been eating into the accuracy of the judgment too. It wasn’t free, though. Like the hub-note exception, the eye that reads all the notes had to be taught a new judgment the candidate filter never needed: what counts as a normal summary and what counts as forbidden duplication. The price of removing the filter wasn’t speed; it moved into teaching that judgment more precisely.