SPOT-Bench v4: how we measure match quality without grading our own homework
Every vendor in this category publishes an accuracy number. Almost none of them publish what it was measured on, which makes the number decorative.
The three ways an accuracy claim goes wrong
- Clean inputs. Benchmarking on register names matched against register names measures string equality. Real inputs are truncated, mangled and OCR'd.
- Self-labelling. If the ground truth comes from your own output, you are measuring self-consistency.
- Selective reporting. Excluding sole traders, or young companies, or non-Latin scripts, moves the headline several points and is invisible unless stated.
v4 is our attempt to be un-embarrassing on all three, and to make it checkable rather than take-our-word-for-it.
Blind adjudication
Two annotators label each record independently against the register of record. Disagreements go to a third adjudicator who never sees either original label. Agreement before adjudication is κ = 0.91, and the 9% that needed adjudication is flagged hard — it is the most useful subset in the whole benchmark.
1,840 records could not be decided even by the adjudicator. They are marked unlabelable and excluded, and we report that they exist rather than quietly dropping them.
Corruptions, at production rates
The corruption generators are in the public harness. The rates are not invented; they are measured against a sample of real traffic, which is why the OCR substitution rate applies only to the OCR subset.
| Corruption | Rate |
|---|---|
| Legal-form mangling | 23.7% |
| OCR substitution (OCR subset) | 14.1% |
| Truncation at a field boundary | 11.2% |
| Diacritic loss | 8.4% |
| Address glued into the name | 6.9% |
The parts where we lose
97.3% top-1 is the aggregate. The distribution underneath is more useful to anyone deciding whether to buy this.
- Sole traders: 84.2%. There is no register key to anchor on, and often no register entry at all.
- Companies incorporated in the last 24 hours: 61.4%. The gazette has not printed.
- Transliterated non-Latin names: 91.8%.
- Same-name companies with no geography hint: 88.9%, where we deliberately return both rather than pick one.
If your data is mostly sole traders, the number that matters to you is 84.2%, not 97.3%. We would rather you knew that before signing than after.
Run it yourself
pip install spot-bench, point it at a test key, and the public 10,000-record subset runs in a few minutes. The adapter interface is three methods, so it runs against anyone else's resolver too — which is the point. Details in the methodology.
Try it against your own data
The Developer plan is free forever and needs no card. 2,500 credits is enough to answer the only question that matters: does it resolve your records.