Gemini 4 Argon is a surprise on transcription and it requires more analysis and explanation than normal. Headline is that its’s clearly SOTA: modified WER of 1.73% and CER of 0.64% on low effort. But that’s not the whole story because essentially the benchmark is not saturated. First of all, Argon works a bit differently. It follows instructions better (it successfully differentiates and ignores archival markings, page numbers etc) and it also does better with unusually formatted pages and text, like the attached image. Most importantly its errors are also far less consequential… and I’m not sure we should call them errors. When the initial results came back, they felt much better than the raw numbers suggested. At first it was scoring about the same as 3.1 pro but the transcriptions simply looked better. So we did a deep dive and when we examined all 174 word errors, we found that all but 40 were essentially toss-ups. These were all cases where the original was misspelled but where it’s also unclear what the correct, ground truth misspelling should be. Basically squinting and arguing about whether that clearly incorrect letter is an erroneous e, o, or a. The point is these do t really matter because the original is unclear. This hadn’t really been an issue on previous tests because most of the error were still meaningful or at least represented things like a misspelled word in the original that had been corrected. This was different. When you put it all together, we basically have a saturated benchmark, at least on 18th and 19th century English text. Gemini 4 Argon’s effective word error rate is something like 0.4% and it’s effective CER is about 0.2%. I am being imprecise because people will disagree about how we should handle these types of residual errors. What is clear, is that on our benchmark anyway, Argon is much better than human experts. It’s also slightly cheaper to use than 3.1 pro and even a bit faster, in our testing. This is a significant moment because basic textual accuracy is the precursor to everything else historians want to do with documents (and any other knowledge work which involves historical documents or handwritten papers). Once you have accurate text, you can start to actually envision scaling out massive research projects. And it looks like we’re there. Sorry this was a bit late (people have been asking), it’s been a very busy week and I wanted to get this right.