SPOT-Bench v4 methodology

A vendor grading its own homework is worth exactly nothing. This is how SPOT-Bench is built, what it does not measure, and how to reproduce the number on your own machine.

Where the records come from

250,000 input records drawn from four real sources, in the proportions we see in production rather than in proportions that flatter us.

Labelling

1. Two annotators label each record independently against the register of record, not against our output.
2. Disagreements go to a third adjudicator who sees neither original label.
3. Inter-annotator agreement before adjudication is κ = 0.91. The 9% that needed adjudication is the interesting part of the set and is flagged as hard in the labels.
4. Records where the adjudicator could not decide are marked unlabelable and excluded from the score — 1,840 of them, 0.7%.

The five corruptions

Real inputs are damaged in specific, repeatable ways. Each is applied at the rate we observe in production, and the generators are in the harness so you can inspect the rates.

• Truncation — a 40-character CRM field cutting a name mid-word. 11.2% of records.
• Legal-form mangling — GmbH & Co. KG becoming GMBH CO KG, or vanishing entirely. 23.7%.
• Diacritic loss — round-tripping through a system that is not UTF-8 clean. 8.4%.
• Address glue — the address concatenated into the name field. 6.9%.
• OCR substitution — rn for m, 0 for O, at character error rates measured from real scans. 14.1% of the OCR subset.

What we report

Where we lose, in full

• Sole traders and unregistered businesses: 84.2% top-1. No register key exists to anchor the match.
• Companies registered within the last 24 hours: 61.4%. The gazette has not printed yet.
• Non-Latin scripts transliterated and back: 91.8%.
• Deliberately similar names in the same jurisdiction with no geography hint: 88.9%, and we return both candidates rather than pick.

A benchmark that only contains the cases you win is a marketing asset, not a measurement.

Running it yourself

The harness, the corruption generators and the label schema are public. The labelled set itself is available under a research agreement — not to be difficult, but so that it does not end up inside a training corpus and stop measuring anything.

$ pip install spot-bench
$ export SPOTIT_KEY=spk_test_8f2c...

# the public subset, 10,000 records, no agreement needed
$ spot-bench run --subset public --provider spotit

# or point it at any other resolver implementing the adapter interface
$ spot-bench run --subset public --provider ./my_adapter.py

The adapter interface is three methods. We would genuinely rather you ran it against a competitor as well; a benchmark with one participant is a brochure. Request the full set from press@spotit.ai.

What it does not measure

• Coverage. A resolver cannot match what is not in its database. Coverage is reported separately, per jurisdiction, on the resources page.
• Freshness. SPOT-Bench is a static set; latency to a register event is measured continuously and published on the status page.
• Your data. Our corruptions are drawn from our traffic. Run the harness against your own file before believing any of it — bring a sample.