OpenAI's Decisions API was fast, but it didn't fit my merge criteria

I’m building an Obsidian plugin that isn’t public yet. You feed it source material, and an LLM reads it and organizes it into notes, one per concept. As material piles up, sometimes two notes end up covering the same thing, and each time the plugin sends both notes to the OpenAI API (gpt-6-luna) and asks: “Is this the same concept, so merge them, or different concepts, so keep them apart?” That’s a full LLM call to get a single answer, so I’ve been checking whether a model built only for this kind of judgment could take over.

In the previous post I tried Laya, an open-source reproduction of the decision-only model Jev. This time I put TypeSafe’s Jev itself, whose account opened for me after that post, and the OpenAI Decisions API, released in public beta on October 6, on the same problem. The short version: the Decisions API was the fastest of the three and gave the exact same answer every time. But under the judgment criteria my plugin actually uses, it attached high confidence even to wrong answers. It’s less that the tool is bad and more that the shape of the question I asked didn’t fit it.

On the easy problems, Jev and gpt-6-luna both scored perfectly

The test data came in two sets. One was the 23 pairs from the Laya test in the previous post: 17 genuine duplicates created twice from the same article, plus 6 synthetic pairs that only share a title. The other was 40 note pairs the plugin had actually checked as duplicate candidates (28 in Korean, 12 in English). I read both notes of each of the 40 pairs and set the answer myself: 15 pairs were the same concept, 18 were different, and 7 were ambiguous, where reasonable calls could differ.

When I ran Jev and gpt-6-luna on this data in early October, both got all 23 pairs right, and all 33 pairs with a clear answer. The differences were speed and confidence. Jev took about 0.27 seconds per judgment, gpt-6-luna about 1.3 seconds. Jev tends to attach low confidence when it’s unsure, so if I only followed answers with confidence 0.8 or higher, it handled 29 of the 33 pairs with no errors. Only the remaining 4 would need to go to the LLM.

OpenAI Decisions API: 0.13 seconds, the same answer every time

The Decisions API is a new OpenAI endpoint that doesn’t generate text like an LLM; it only answers predefined questions in a predefined format. It answers in three forms: a true/false probability (predicate), one of several options (choice), or a graded score (score), and the only model is gpt-6-luna. According to the announcement, it’s up to 10x faster than calling gpt-6-luna through the Responses API, costs $0.10 per million input tokens, and doesn’t charge for output tokens. It’s OpenAI’s own entry in the same “decision-only” category as Jev.

With the same question and the same option descriptions, a judgment took about 0.13 seconds: half of Jev, a tenth of a regular call. I asked each pair three times, and not only the answer but even the confidence value was identical all three times. It got all 23 easy pairs right.

Just reordering the options moved accuracy from 16/33 to 32/33 (option order bias)

The 40 pairs produced something odd. With the options in the order “same / different,” it answered “same” on 39 of the 40 pairs and got only 16 of the 33 clear pairs right. Asked again with only the option order flipped to “different / same,” it got 32 right. The “merge / keep separate” question likewise went from 31 to 33. Pairs whose answer was “same” were right in either order, so this was a pull toward the merge-side option whenever it came first.

This bias isn’t unique to the Decisions API. In the document where TypeSafe lists Jev 1.13’s known weak spots, it writes that “the order of a Choice’s options can affect the answer, and jev-1.13 leans toward the option that comes first,” and recommends reordering the options to check that the answer holds. The fix is simple: put the conservative option (“keep separate”) first, or ask in both orders and only act when the answers agree. The Decisions API is fast enough that asking twice is still quicker than asking Jev once.

It turned out I wasn’t asking the plugin’s question (LLM benchmark)

While re-deciding the answers for the ambiguous pairs, I found a more fundamental problem. The plugin’s “is this the same concept?” prompt contains fairly specific rules. If two notes from different sources each explain the same general topic (say, psychological safety) with different examples, they count as different concepts. If they come from different articles covering the same event, they’re the same concept. If one is a subtopic of the other, they’re not the same concept. And when in doubt, keep them apart. These rules were added while sorting out earlier cases where merge decisions had gone back and forth.

None of these rules were in the questions I’d been asking the three models. I’d only been asking “is this the same concept?” in the common-sense sense. The numbers above were scores on “duplicate detection in general,” not on the judgment the plugin actually makes. So I re-decided the ambiguous pairs under those rules, and sent the three models the plugin’s prompt word for word, in the same message format. For gpt-6-luna I set reasoning effort to high, the setting I actually use.

Re-running with the plugin’s own prompt: confidence 0.94 on a wrong answer (confidence calibration)

Here are the results of running all 40 pairs three times.

By correct count alone, the gap isn’t large. What separated them was how they looked when wrong. Jev attached low confidence, around 0.1 to 0.5, to most of its wrong answers. Following only answers with confidence 0.9 or higher, it handles 22 of 37 pairs (leaving out three pairs whose expected answer is itself doubtful) with no errors, and the rest can go to the LLM. The Decisions API, on the other hand, got two subtopic pairs wrong as “same” with confidence 0.94 and 0.96. Filtering by confidence and sending only the unsure answers to the LLM doesn’t work under these conditions. The option-order bias from earlier also no longer pointed in a consistent direction.

A plugin bug found along the way: notes on the same product weren’t being merged (duplicate note merge)

Two pairs stood out in the results: “Claude Code” and “클로드 코드” (Claude Code written in Korean), and “스페이스X” and “스페이스엑스” (two Korean spellings of SpaceX). Same product, same company, just written differently, yet gpt-6-luna and Jev said “different concept” for both pairs, all three times. The Decisions API also said “different” for the Claude Code pair, with confidence 1.0. When I asked gpt-6-luna for its reasoning, it said (translated from Korean):

Both notes cover the same product, Claude Code, but A describes Uber’s adoption and budget overrun, while B describes … using data on a personal PC. The specific events and facts don’t overlap, so I judge them to be separate concepts.

The model wasn’t wrong; the rule was being misapplied. The “do they cover the same specific event?” criterion was meant for general topics like psychological safety (how that rule came about is in this post). But since its scope wasn’t written down, it was applied to product and company notes too. The same product is the same thing no matter which article it came from. This Claude Code pair had failed to merge once before. Back then the cause was a scope setting that excluded product and company notes from comparison entirely; with that fixed, the pair now reaches the candidate list, only to be blocked at the judgment step this time.

I added an exception to the rule, “a unique entity such as the same product, company, person, or work is the same thing even if the sources or events differ,” and re-ran gpt-6-luna. My first wording fixed the two pairs but had a side effect: it flipped one parent/subtopic pair from the same article to “same.” Putting the exception paragraph at the very top of the rule seemed to push “do the specific facts overlap?” to the front and bury “a subtopic is not the same concept” right below it. When I moved the exception to a separate item after the rule and added “a subtopic relationship is still a different concept,” the two pairs were fixed and the side effect disappeared. Correct answers went from 106 to 111 out of 120. This change is still at the trial stage and hasn’t gone into the plugin.

Is the Decisions API a bad tool? I packed several judgments into one question (prompt design)

It would be hard to call the Decisions API useless from these results alone. Looking back at the shape of the question explains why. What I sent was a single choice question, “is this the same concept?”, but its instructions held several judgments: is it the same unique entity, is one a subtopic of the other, do the specific facts overlap, and which way to lean when unsure. Called through a regular request, gpt-6-luna can work through those conditions one by one with reasoning. The Decisions API’s usage record, on the other hand, showed zero reasoning tokens. It answers in a single shot, without stepping through the conditions in turn.

Both companies’ docs point the same way. OpenAI’s Decisions API guide says “Separate different concerns into different questions,” and TypeSafe’s Jev doc lists “Hiding several judgments inside one question” as a pattern to avoid. It also warns that accuracy falls when the input contains a lot of content unrelated to the decision. My input held the full note text plus metadata and a list of related links.

So this result is closer to “it doesn’t fit my current criteria, which pack several conditions into one question” than to “the Decisions API can’t make judgments.” The direction worth trying is clear: split the judgment into several true/false questions, such as “do both notes cover the same unique entity?”, “is one a subtopic of the other?”, and “do the specific facts overlap?”, and let code combine the answers. Also trim the input down to the text the judgment needs.

The same goes for Jev. Jev can’t be fine-tuned on customer data, and I couldn’t find tuning knobs like temperature in its API docs either. All you can touch are the instructions, the option descriptions, and how the input is split up. So the way to improve Jev further is the same: split the question, trim the input, and check with reordered options. I haven’t measured yet whether this actually helps. That’s the next experiment.

Final thoughts

The biggest lesson this time is that a benchmark only tells you how a model scores on the question you asked. In the first few experiments, all three models looked great. But those were scores on a common-sense “same concept?”, and once I sent the question the plugin actually asks, the picture changed. Along the way, a gap in the plugin’s own rule surfaced too.

And “doesn’t fit” is not the same as “bad.” The Decisions API is fast, gives the same answer to the same question every time, and only charges for input tokens. For a problem that comes down to one clean judgment, it’s a good choice even now. To fit it to my problem, I need to reshape the question to fit the tool. Since it’s in public beta, I plan to look at it again after the official release.