How often it gets this wrong
Every result rests on a judgement a model makes about a snippet it was handed. Models read snippets wrong. Here is how often this one does, including the target it does not meet.
This extension does not tell you whether something is true. It tells you who said it, weighs them against a list you keep yourself, and shows you the sentence it based that on. All of which depends on one judgement being made correctly, over and over: does this search result actually say what the claim says?
That judgement is made by a language model. A model can read a snippet wrong, and this one does. The numbers below are what happened when 223 of those judgements were checked by hand.
What these numbers actually mean
Two different things are being counted here, and the gap between them is not an accident. The 61% is per source: hand the model one search result and one claim, and that is how often it picks the label a human would. The 75% is per claim, after all of that claim's sources have been weighed together. The second number is higher because a verdict survives individual sources being misread. It is also the number that corresponds to something you actually see.
The figure treated as the important one is neither of those.
Telling you a claim is contradicted, when the source does not contradict it, is saying something untrue about the world with a citation attached.
Showing no source found when a source existed wastes your time, and you will notice. The opposite error does not announce itself. So it is tracked on its own, held to a 95% bar rather than folded into an average, and it currently falls short of that bar.
Where it is weakest
Averages hide the shape of a failure. Broken out by label, one row collapses while the rest hold up.
the same numbers as a table
| Label | Precision | Recall | Cases |
|---|---|---|---|
| supports | 0.85 | 0.73 | 55 |
| partial | 0.29 | 0.17 | 35 |
| insufficient | 0.56 | 0.73 | 84 |
| off_topic | 0.53 | 0.57 | 30 |
| contradicts | 0.87 | 0.68 | 19 |
The weak row is partial: a source that settles one half
of a two-part claim and says nothing about the other half. It finds
under a fifth of those, and files most of the rest as
insufficient. That failure at least runs in the safe
direction. It under-claims rather than over-claims, so the visible
result is a claim that looks less well sourced than it really is.
It is still the thing most in need of fixing.
It also reads only titles and search snippets. Not full articles, not PDFs, and nothing behind a paywall. A claim that exists only inside a document nobody has indexed comes back as no source found, which is honest and not much help.
How this was measured
223 real cases, taken from actual usage rather than invented, and labelled by hand one at a time. Labels were never seeded from what a model said. A set labelled that way can only ever confirm the errors it already contains, which makes it worse than no set at all. Where a label turned out to be wrong on review, the case was corrected and marked as corrected. Cases where the right answer is genuinely arguable are kept and flagged rather than quietly dropped.
The verdict figure runs each claim's hand labels and the model's labels through the same aggregation the product uses, with no trust list applied, so what it measures is the judgement step on its own. Your own ratings change which sources carry weight, and can therefore move a verdict in either direction from what is shown here.
Two parts of the pipeline are not covered by any of this: how well a highlighted sentence gets narrowed into a checkable claim, and how good the search results were to begin with. A correct judgement about a poor set of sources is still a poor answer, and nothing on this page measures that.
Why publish it
Because the alternative is asking you to take it on faith, and this is a tool whose whole argument is that you should not have to. A number chosen to flatter would be worth less than a number that falls short of its own target, which is what most of these do.
They are re-measured whenever the prompts change, and this page moves with them, including when they move the wrong way.