Verification metrics for a system that decides whether two things match: a face or voice verifier, a signature checker, a duplicate detector. The input is one similarity score per pair (higher means more alike) and a label per pair, true when the two really are the same (a genuine pair) and false when they are not (an impostor pair).
A pair is accepted when its score is greater than or equal to the threshold. Three rates describe a threshold: - FAR, the false acceptance rate: impostor pairs accepted, divided by impostor pairs. - FRR, the false rejection rate: genuine pairs rejected, divided by genuine pairs. - accuracy: pairs decided correctly, divided by all pairs. Raising the threshold lowers FAR and raises FRR. The equal error rate (EER) is the value where the two are equal.
The curve. Verification->Evaluate sorts the distinct scores v0 < v1 < ... < v(k-1) and reports one operating point per threshold v(j), then a last point at +Infinity that accepts nothing (FAR 0, FRR 1). The first point (threshold v0) accepts everything (FAR 1, FRR 0). So there are k + 1 points, thresholds ascending, FAR never increasing and FRR never decreasing. Pairs that share a score share a point, so a tie is never split.
The EER and its interpolation rule. Let d(j) = FAR(j) - FRR(j); d is 1 at the first point and -1 at the last, and never increases. The EER is read at the first point j with d(j) <= 0. When d(j) is 0 the EER is FAR(j) and the threshold is v(j). Otherwise the two rates cross between point j-1 and point j, and both the EER and the threshold are linearly interpolated along that segment at lambda = d(j-1) / (d(j-1) - d(j)): EER = FAR(j-1) + lambda * (FAR(j) - FAR(j-1)), threshold = v(j-1) + lambda * (v(j) - v(j-1)). When the crossing lies after the highest score (the segment ends at +Infinity) the threshold reported is the highest score, and the EER is still interpolated. The same rule is what scipy's interp1d over a ROC curve gives, which is how the regression test checks it.
No call here ends the program on bad input. Evaluate returns a result whose IsValid() is false and whose GetError() says why (a size mismatch, an empty input, a NaN or infinite score, no genuine pair or no impostor pair); every number in it is then NaN. The single-number functions return NaN for the same input; call Verification->Check to read the reason. GetError() is "" when there is nothing to report. Needs only gen_collect: compile with -lib ml_verify,gen_collect.
Verification
Verification metrics from scores and same / different labels. Every function is static; see the bundle description for the rules (accept when score >= threshold, FAR, FRR, the curve, the EER interpolation) and for how bad input is reported.
Operations
Accuracy # function
The accuracy at one threshold: pairs decided correctly over all pairs.
function : Accuracy(scores:Float[], genuine:Bool[], threshold:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| scores | Float | one similarity score per pair |
| genuine | Bool | true for a same pair |
| threshold | Float | the threshold |
Return
| Type | Description |
|---|---|
| Float | accuracy in 0 to 1, or NaN on invalid input (see Check) or a NaN threshold |
Check # function
Why a set of scores and labels cannot be evaluated.
function : Check(scores:Float[], genuine:Bool[]) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| scores | Float | one similarity score per pair, higher meaning more alike |
| genuine | Bool | one label per pair: true for a same pair, false for a different pair |
Return
| Type | Description |
|---|---|
| String | "" when the input is usable, otherwise the reason |
Evaluate # function
Evaluates a verifier: the full FAR / FRR curve, the EER and its threshold, from which accuracy, the best threshold and the ROC and DET point lists are read.
function : Evaluate(scores:Float[], genuine:Bool[]) ~ VerificationResultParameters
| Name | Type | Description |
|---|---|---|
| scores | Float | one similarity score per pair, higher meaning more alike; must be finite |
| genuine | Bool | one label per pair: true for a same pair, false for a different pair; both kinds must occur |
Return
| Type | Description |
|---|---|
| VerificationResult | a result; check IsValid() (when false GetError() says why) |
Example
r := Verification->Evaluate(scores, same);
if(r->IsValid()) {
"EER {$r->GetEer()} at threshold {$r->GetEerThreshold()}"->PrintLine();
};Far # function
The false acceptance rate at one threshold: impostor pairs with a score at or above it, over all impostor pairs.
function : Far(scores:Float[], genuine:Bool[], threshold:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| scores | Float | one similarity score per pair |
| genuine | Bool | true for a same pair |
| threshold | Float | the threshold |
Return
| Type | Description |
|---|---|
| Float | FAR in 0 to 1, or NaN on invalid input (see Check) or a NaN threshold |
Frr # function
The false rejection rate at one threshold: genuine pairs with a score below it, over all genuine pairs.
function : Frr(scores:Float[], genuine:Bool[], threshold:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| scores | Float | one similarity score per pair |
| genuine | Bool | true for a same pair |
| threshold | Float | the threshold |
Return
| Type | Description |
|---|---|
| Float | FRR in 0 to 1, or NaN on invalid input (see Check) or a NaN threshold |