This series measures whether the LLM judgments in the Obsidian plugin I’m building (not yet public) can be handed to decision-only models. The candidates are TypeSafe’s Jev and OpenAI’s Decisions API. Both pick one of a set of predefined options and attach a confidence value, and at around 0.2 seconds per call, they are more than ten times faster than a regular LLM call. In the previous post, for the judgment “are these two notes the same concept,” splitting the question into three true/false questions made the Decisions API as accurate as gpt-6-luna called the regular way.
This time it’s a different judgment. The plugin has a step where an LLM deletes duplicate sentences between notes, and after it deletes sentences, the remaining sentences kept breaking. “This technology stack,” which pointed to an earlier sentence, loses what it points to; a sentence starting with “Also, …” is left on its own; or only part of a list remains and the meaning changes. I wrote up how that happened in another post. The conclusion was to have an LLM reread the edited note and judge “did this edit create a new problem?”, and the judgment made with gpt-6-luna was fairly accurate. It takes about 3 seconds per note. Could a decision-only model make this judgment in 0.2 seconds, too?
What I measured with — an LLM-as-a-judge benchmark
I collected 134 notes that an LLM had edited while removing duplicates, from the plugin’s real tests and from repro experiments. For each note, I feed in the before and after versions together with the changed paragraphs. I read every one myself to set the answers: 20 notes that actually broke, 11 ambiguous ones, and 103 normal ones.
I compared three approaches.
- A regular gpt-6-luna call: reasoning high, as the plugin is configured. It’s asked with a prompt that spells out the four judging criteria.
- One question: a choice question, “no problem / problem,” built on the same criteria. Earlier in the series the answers had leaned depending on option order, so I asked in both orders.
- Split into four questions: ask separately, as true/false, whether a pointing word lost its referent, whether an incomplete sentence appeared, whether the meaning changed, and whether a section heading stopped matching its content, and treat it as a problem if any one is true. This is the approach that rescued the Decisions API in the previous post.
Results — the decision-only models flagged about four times as many normal notes (false positive rate)
| Approach | Broken notes caught | Normal notes wrongly flagged | Time per call |
|---|---|---|---|
| gpt-6-luna | 18/20 | 3/103 | ~2.9 s |
| Decisions API, one question | 15–16/20 | 12–18/103 | ~0.25 s |
| Jev, one question | 14–17/20 | 10–12/103 | ~0.17 s |
| Decisions API, four questions | 19/20 | 74/103 | ~0.26 s |
| Jev, four questions | 18/20 | 49/103 | ~0.17 s |
Asked as one question, both models caught fewer broken notes and flagged about four times as many normal notes as gpt-6-luna. Since I planned to roll flagged notes back to how they were before the edit, flagging a normal note means throwing away dedup that should have happened.
Splitting the question made it worse (Decisions API predicate · Jev noul)
In the previous post, the Decisions API only got it right once the question was split. This time it was the opposite. It caught nearly all the broken notes, but flagged more than half of the normal ones.
Looking at which questions were the problem, there were two. Of the 103 normal notes, the Decisions API said true for “did a section heading stop matching its content” on 64, and for “did the meaning change” on 48. By contrast, “did a pointing word lose its referent” was close to 0 on normal notes, and came out near 1.0 on notes that had actually lost their referent, like “this technology stack” and “the shortlist.” The question asked with a precise criterion did its job, and the two loose questions fired on almost every note.
The reason is an unverified guess, but I have an idea. Deleting duplicate sentences makes sections shorter and narrows what a note covers. Read literally, “did a section heading stop matching its content” and “did the meaning change” are somewhat true of most notes that had duplicates removed. What I wanted to ask was “apart from removing the duplicate, did that create a new problem?” gpt-6-luna made that distinction while reasoning, and the decision-only models, which answer in one shot, took the questions as written. It’s the same shape as in the previous post, where “do they summarize the same specific event?” was read as “is it the same article?”
Filtering by confidence saved little (confidence threshold)
The setup that was worth using earlier in the series was “use the decision-only model’s answer only when it’s highly confident, and pass the rest to gpt-6-luna.” Jev’s tendency to be unsure of itself when it’s wrong held this time too. When asked as one question, the highest confidence on a wrong answer was 0.74 or 0.82, depending on option order. The Decisions API had two wrong answers at 0.88 in one of the orders.
The problem was that there were few high-confidence answers to begin with. Following only answers with confidence 0.8 or higher, the number of broken notes caught was the same as gpt-6-luna alone, and the normal notes wrongly flagged were the same or two more, but the decision-only model handled only 15 to 29 of the 134 notes on its own. That cuts gpt-6-luna calls by only 11–22%. Raising the confidence threshold to 0.9 shrank even that to 6 to 15 notes.
Only a few notes get edited in one round of note dedup, and even with gpt-6-luna it’s about 3 seconds per note. The complexity of attaching one more model is large compared to the time saved. So I decided to make this judgment with gpt-6-luna alone.
Closing thoughts
The previous post ended with “splitting the question made it work.” This time, when I carried the same method over to a different judgment as is, it flagged more than half of the normal notes. What worked in the previous post wasn’t the method of “splitting the question” itself, but that each split question was written so it could only be read the way I meant it. Two of the four I split this time weren’t.
The two judgments were different in nature, too. “Are these two notes the same concept” is a judgment made by looking at the whole note. “Did the edited note get a broken sentence” requires finding one problem sentence inside a long note, and then telling whether that problem is a normal result of dedup or not. Expecting the decision-only models to match an LLM on the second judgment because they did on the first was assuming that the same tool would give the same result. Each judgment had to be measured separately.