Why we publish how we test, false alarms included

When a hiring team asks me whether Verified Call works, I think the honest answer has two parts: how often it finds the problem, and how often it is wrong about a person who did nothing wrong. This post explains what we publish for each test, and why.

A catch rate is half an answer

Any check can catch every face swap if it flags everyone. That check would have a perfect catch rate and be useless. The number that tells you whether you can trust a flag is the other one: how often the check raises an alarm on a real person.

In hiring, that matters more than in most places. A false alarm does not land on a file. It lands on a candidate who is trying to get a job, often nervous, sometimes working in a second language. So we put the false alarms right next to the catches, measured the same way. We wrote more about this in why you should ask any interview integrity tool for its false-alarm rate.

What we publish for each test

Each result on our Research page carries the same parts:

1. What we did, in plain words. For example: we opened each of 4 free AI answer tools 30 times and checked that Verified found it each time.
2. The result as a count, such as 30 of 30 tries, so you can see the sample size.
3. The false alarms, counted the same way, such as 0 of 30 tries with no AI tool open.
4. The sample: what was tested, and how many people, tools or cameras.
5. The date the test was run.
6. How sure we can be, as the lowest rate the result supports at 95% confidence.
7. Live or recorded: whether the test ran on live calls or on recordings tested offline.

What that looks like

Here are four results as they appear on the research page. The right-hand column is the one most vendors leave out.

Not every number is a clean sweep, and that is the point. The same-speaker results come from public recordings of phone calls in 8 Indian languages, from 1,134 people, tested offline.

Small samples, said plainly

A result like 30 of 30 sounds perfect. It is not proof of perfect. Thirty tries is a small sample, so next to each count we show the lowest rate that result supports.

With 30 tries per tool, we are 95% sure it catches each tool at least 90.5% of the time.

That sentence is less exciting than "30 of 30". It is also more useful. It tells you how much weight the result can carry, and it tells us where more testing would help.

“Can’t confirm” is its own answer

In the face test, none of the 3,990 sessions with someone else's face got through. But they did not all end the same way: 2,855 were flagged, and 1,135 ended "Can't confirm". We show those two numbers separately instead of folding them into one catch rate.

The host sees the same difference in a call. "Can't confirm" means the check could not confirm it. It is not an accusation, and the host can simply ask about it.

What is not in the beta is labelled too

The research page has a "Coming up" list: what we are building and researching now, and how far along each piece is. Those items are not in the beta yet, and the page says that in plain words. A test result from research is not a product feature until it ships.

What to ask any vendor

You do not need to read a methods paper to judge a check. Ask four things, of us or anyone else:

• How often does it catch the problem, as a count?
• How often does it raise an alarm on someone who did nothing wrong, measured the same way?
• What was tested, on how many people or devices, and when?
• Was it a live call or a recording?

If a vendor can answer all four, you can compare them fairly. If they can only answer the first, you have half an answer.