[Complete] Series: I Almost Mistook Two Variables for One
Same sources, same plugin, but switching machines made the results differ 3x. Tracking down the cause, what I thought was a single culprit turned out to be two different settings, and separating them took rerunning the experiment eight times; along the way I found and fixed three completely unrelated bugs.
-
189 notes an LLM had organized became 63 one day
I had an LLM turn the same 30 sources into Obsidian notes, and just switching machines cut them from 189 to 63. Not one cause, chunk size, but two settings.
-
Just lowering the LLM chunk size made the fidelity problems go away
I compared the same 30 sources changing only the chunk size, how many go to the LLM at once. Summary errors that drop people and quotes fell from 40% to 0%.
-
Getting ready to rerun the experiment, git conflict markers got committed as-is
Continuing a chunk-size experiment at home, alternating two laptops on one git repo committed conflict markers into 77 files. Then I suspected reasoning effort.
-
OpenAI reasoning effort wasn't the culprit behind the fidelity problems
Lowering OpenAI reasoning effort from High to Medium didn't add LLM summary errors. Note count and run time changed a lot instead; chunk size governed accuracy.
-
I used embedding similarity to find out why Korean and English notes wouldn't merge
Korean and English notes on one concept never merged. OpenAI embeddings put them at 0.77, below same-language duplicates; a 0.70 similarity threshold fixed it.
-
Raising LLM request concurrency from 2 to 4 surfaced two other bugs
Raising parallel LLM requests from 2 to 4 kept summary accuracy. The audit found missed Korean-English note merges and Korean headings on English notes instead.
-
Laptop 2's OpenAI reasoning effort turned out to be Medium
The work laptop's OpenAI reasoning effort was the default Medium, confirming chunk size alone caused the LLM summary errors. The two are separate knobs.