Verification metrics for a system that decides whether two things match: a face or voice verifier, a signature checker, a duplicate detector. The input is one similarity score per pair (higher means more alike) and a label per pair, true when the two really are the same (a genuine pair) and false when they are not (an impostor pair).
A pair is accepted when its score is greater than or equal to the threshold. Three rates describe a threshold: - FAR, the false acceptance rate: impostor pairs accepted, divided by impostor pairs. - FRR, the false rejection rate: genuine pairs rejected, divided by genuine pairs. - accuracy: pairs decided correctly, divided by all pairs. Raising the threshold lowers FAR and raises FRR. The equal error rate (EER) is the value where the two are equal.
The curve. Verification->Evaluate sorts the distinct scores v0 < v1 < ... < v(k-1) and reports one operating point per threshold v(j), then a last point at +Infinity that accepts nothing (FAR 0, FRR 1). The first point (threshold v0) accepts everything (FAR 1, FRR 0). So there are k + 1 points, thresholds ascending, FAR never increasing and FRR never decreasing. Pairs that share a score share a point, so a tie is never split.
The EER and its interpolation rule. Let d(j) = FAR(j) - FRR(j); d is 1 at the first point and -1 at the last, and never increases. The EER is read at the first point j with d(j) <= 0. When d(j) is 0 the EER is FAR(j) and the threshold is v(j). Otherwise the two rates cross between point j-1 and point j, and both the EER and the threshold are linearly interpolated along that segment at lambda = d(j-1) / (d(j-1) - d(j)): EER = FAR(j-1) + lambda * (FAR(j) - FAR(j-1)), threshold = v(j-1) + lambda * (v(j) - v(j-1)). When the crossing lies after the highest score (the segment ends at +Infinity) the threshold reported is the highest score, and the EER is still interpolated. The same rule is what scipy's interp1d over a ROC curve gives, which is how the regression test checks it.
No call here ends the program on bad input. Evaluate returns a result whose IsValid() is false and whose GetError() says why (a size mismatch, an empty input, a NaN or infinite score, no genuine pair or no impostor pair); every number in it is then NaN. The single-number functions return NaN for the same input; call Verification->Check to read the reason. GetError() is "" when there is nothing to report. Needs only gen_collect: compile with -lib ml_verify,gen_collect.
VerificationResult
The result of Verification->Evaluate: the whole FAR / FRR curve and the numbers read from it. Immutable once built. See the bundle description for the threshold rule and the EER interpolation. Nothing here ends the program: on a failed evaluation IsValid() is false, GetError() says why and every number is NaN (arrays are Nil).
Operations
- AccuracyAt
- FarAt
- FrrAt
- GetBestAccuracy
- GetBestAccuracyThreshold
- GetCurve
- GetDetCurve
- GetEer
- GetEerThreshold
- GetError
- GetFarCurve
- GetFrrAtFar
- GetFrrCurve
- GetGenuineCount
- GetImpostorCount
- GetRocCurve
- GetThresholdAtFar
- GetThresholds
- IsValid
- Size
AccuracyAt #
The accuracy at a threshold: pairs decided correctly over all pairs.
method : public : AccuracyAt(threshold:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| threshold | Float | the threshold |
Return
| Type | Description |
|---|---|
| Float | accuracy in 0 to 1, or NaN when the evaluation failed or the threshold is NaN |
FarAt #
The false acceptance rate at a threshold (a pair is accepted when its score is greater than or equal to it), from the stored curve.
method : public : FarAt(threshold:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| threshold | Float | the threshold |
Return
| Type | Description |
|---|---|
| Float | FAR in 0 to 1, or NaN when the evaluation failed or the threshold is NaN |
FrrAt #
The false rejection rate at a threshold, from the stored curve.
method : public : FrrAt(threshold:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| threshold | Float | the threshold |
Return
| Type | Description |
|---|---|
| Float | FRR in 0 to 1, or NaN when the evaluation failed or the threshold is NaN |
GetBestAccuracy #
The best accuracy reached at any threshold (see GetBestAccuracyThreshold).
method : public : GetBestAccuracy() ~ FloatReturn
| Type | Description |
|---|---|
| Float | accuracy in 0 to 1, or NaN when the evaluation failed |
GetBestAccuracyThreshold #
The threshold with the best accuracy among the observed scores and +Infinity (accept nothing). A tie goes to the lowest threshold.
method : public : GetBestAccuracyThreshold() ~ FloatReturn
| Type | Description |
|---|---|
| Float | the threshold, or NaN when the evaluation failed |
GetCurve #
The curve as one matrix, one row per operating point: [threshold, FAR, FRR], thresholds ascending. The last row has threshold +Infinity.
method : public : GetCurve() ~ Float[,]Return
| Type | Description |
|---|---|
| Float | a (points x 3) matrix, or Nil when the evaluation failed |
GetDetCurve #
The DET curve as rows of [FAR, FRR], FAR ascending, for a plot of FRR against FAR (usually on logarithmic axes; drop the rows where a rate is 0 first).
method : public : GetDetCurve() ~ Float[,]Return
| Type | Description |
|---|---|
| Float | a (points x 2) matrix, or Nil when the evaluation failed |
GetEer #
The equal error rate, in 0 to 1, interpolated as the bundle description states.
method : public : GetEer() ~ FloatReturn
| Type | Description |
|---|---|
| Float | the EER, or NaN when the evaluation failed |
GetEerThreshold #
The threshold at which FAR and FRR are equal, interpolated as the bundle description states.
method : public : GetEerThreshold() ~ FloatReturn
| Type | Description |
|---|---|
| Float | the EER threshold, or NaN when the evaluation failed |
GetError #
Why the evaluation failed.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or "" when it succeeded |
GetFarCurve #
The false acceptance rate at every operating point, in the order of GetThresholds (never increasing, from 1 to 0).
method : public : GetFarCurve() ~ Float[]Return
| Type | Description |
|---|---|
| Float | the FAR values, or Nil when the evaluation failed |
GetFrrAtFar #
The smallest FRR reachable while FAR stays at or under a budget.
method : public : GetFrrAtFar(max_far:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| max_far | Float | the false acceptance budget, in 0 to 1 |
Return
| Type | Description |
|---|---|
| Float | FRR in 0 to 1, or NaN when the evaluation failed or the budget is not in 0 to 1 |
GetFrrCurve #
The false rejection rate at every operating point, in the order of GetThresholds (never decreasing, from 0 to 1).
method : public : GetFrrCurve() ~ Float[]Return
| Type | Description |
|---|---|
| Float | the FRR values, or Nil when the evaluation failed |
GetGenuineCount #
How many genuine pairs were scored.
method : public : GetGenuineCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | the count (0 when the evaluation failed) |
GetImpostorCount #
How many impostor pairs were scored.
method : public : GetImpostorCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | the count (0 when the evaluation failed) |
GetRocCurve #
The ROC curve in the layout Metrics->RocCurve returns and Axes->RocCurve in System.Plot draws: rows of [threshold, FAR, 1 - FRR] with FAR ascending from 0 to 1, starting at the point that accepts nothing (its threshold is the highest score, so every value is finite).
method : public : GetRocCurve() ~ Float[,]Return
| Type | Description |
|---|---|
| Float | a (points x 3) matrix, or Nil when the evaluation failed |
GetThresholdAtFar #
The lowest threshold whose FAR is at most a budget, which is the one with the smallest FRR under that budget ("FRR at FAR = 0.1%").
method : public : GetThresholdAtFar(max_far:Float) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| max_far | Float | the false acceptance budget, in 0 to 1 |
Return
| Type | Description |
|---|---|
| Float | the threshold (+Infinity when only rejecting everything fits), or NaN when the evaluation failed or the budget is not in 0 to 1 |
GetThresholds #
The threshold of every operating point, ascending. The last is +Infinity (accept nothing).
method : public : GetThresholds() ~ Float[]Return
| Type | Description |
|---|---|
| Float | the thresholds, or Nil when the evaluation failed |