Accuracy # native

Calculates accuracy: (TP + TN) / Total

function : native : Accuracy(predictions:Bool[], actuals:Bool[]) ~ Float

Parameters

Return

AtThreshold # function

Recall, false-positive rate and precision at ONE given threshold. The counterpart to RecallAtFpr. That one searches for the operating point fitting a false-alarm budget; this one is handed a threshold and reports what it actually does. Both are needed to answer whether a threshold chosen on one set still honours its budget on another: the budget is intended, the rate reported here is what happened. An instance counts as predicted positive when its score is >= threshold. That matches OperatingPoints, which accumulates from the highest score down and lets instances sharing a score share an operating point, so a threshold from ThresholdAtFpr can be passed straight here. Every rate is guarded. A float division by zero raises in Objeck and would take the program down, and the degenerate cases are ordinary: a threshold above every score predicts nothing positive, leaving precision with no denominator.

function : AtThreshold(scores:Float[], actuals:Bool[], threshold:Float) ~ ThresholdResult

Parameters

Return

AucRoc # function

Area under the ROC curve, by trapezoid over the operating points.

function : AucRoc(scores:Float[], actuals:Bool[]) ~ Float

Parameters

Return

AveragePrecision # function

Average precision: the precision-recall curve summarised as the sum of precision at each point weighted by the recall it gained. Preferred over AUC-ROC when positives are rare, because it does not credit a model for the vast number of true negatives.

function : AveragePrecision(scores:Float[], actuals:Bool[]) ~ Float

Parameters

Return

Bootstrap # function

A bootstrap percentile confidence interval for a metric. Resamples the scored instances WITH REPLACEMENT reps times, recomputes the metric on each resample, and takes percentiles of that distribution. The point estimate is the metric on the original sample, not the mean of the resamples. Resampling pairs, not scores: a resample draws (score, label) together, so the class balance moves between resamples the way it would between samples. That is the variability being estimated, and it matters most where it is largest -- a false-alarm budget of 0.001 on a 10,000-row test set is about ten false positives, and ten is a count whose uncertainty is not negligible. A resample can contain only one class, which leaves AUC undefined. The metrics here report that as a NEGATIVE value rather than a real score, so those resamples are discarded rather than averaged in -- folding a sentinel into the distribution would drag the interval toward it. The number that survived is reported, because an interval built from a small fraction of the requested resamples is not the interval that was asked for.

function : Bootstrap(metric:ScoreMetric, scores:Float[], actuals:Bool[], reps:Int, seed:Int, level:Float) ~ ConfInterval

Parameters

Return

Bootstrap # function

A bootstrap interval at the conventional 95% level.

function : Bootstrap(metric:ScoreMetric, scores:Float[], actuals:Bool[], reps:Int, seed:Int) ~ ConfInterval

Parameters

Return

ConfusionMatrix # native

Creates a confusion matrix

function : native : ConfusionMatrix(predictions:Bool[], actuals:Bool[]) ~ Int[,]

Parameters

Return

F1Score # native

Calculates F1 score: 2 * (Precision * Recall) / (Precision + Recall)

function : native : F1Score(predictions:Bool[], actuals:Bool[]) ~ Float

Parameters

Return

MCC # native

Calculates Matthews Correlation Coefficient (MCC)

function : native : MCC(predictions:Bool[], actuals:Bool[]) ~ Float

Parameters

Return

OperatingPoints # function

Operating points of a scored classifier, one row per distinct score, from the highest score down: [threshold, false positives, true positives]. Every metric below is a walk over this. It exists because the functions above take already-thresholded Bool predictions, which throw away exactly what these questions need: a single Bool array fixes one operating point, and cannot answer what the recall would be at a different false-alarm rate. Instances that share a score share an operating point. Splitting them would invent an ordering the scores do not express and let a metric read better than the model earned.

function : OperatingPoints(scores:Float[], actuals:Bool[]) ~ Float[,]

Parameters

Return

Percentile # function

A percentile of a set of values, by linear interpolation. Used for bootstrap intervals, and exposed because a caller resampling a statistic this bundle does not implement still needs the interval. p is clamped rather than rejected: 0.0 gives the minimum and 1.0 the maximum, which is what a caller asking for them wants.

function : Percentile(values:Float[], p:Float) ~ Float

Parameters

Return

PrCurve # function

The precision-recall curve as [threshold, recall, precision] rows.

function : PrCurve(scores:Float[], actuals:Bool[]) ~ Float[,]

Parameters

Return

Precision # native

Calculates precision: TP / (TP + FP)

function : native : Precision(predictions:Bool[], actuals:Bool[]) ~ Float

Parameters

Return

PrecisionAtBaseRate # function

Precision an operating point would show at an assumed base rate. A detection rate and a false-alarm rate are properties of the detector. The precision an operator sees is not: it depends on how common the thing being detected actually is. The same detector that looks precise on a benchmark where half the records are positive can raise mostly false alarms in a deployment where one in a thousand is. precision = tpr*pi / (tpr*pi + fpr*(1-pi)) Worth being blunt about what this does NOT do. It reprojects the measured rates onto a different class balance, assuming both rates stay the same. They often do not: a deployment whose base rate differs usually differs in other ways too, and the detector does not have to behave identically. The number answers "if these rates held, what precision would this base rate imply" -- which is a bound worth knowing, not a prediction.

function : PrecisionAtBaseRate(tpr:Float, fpr:Float, base_rate:Float) ~ Float

Parameters

Return

PrecisionAtBaseRate # function

Precision a measured operating point would show at an assumed base rate. Takes the rates from a ThresholdResult, so a threshold that has actually been measured on held-out data can be reprojected without the caller restating its numbers. The same caveat applies: this assumes both rates carry over.

function : PrecisionAtBaseRate(result:ThresholdResult, base_rate:Float) ~ Float

Parameters

Return

PrintConfusionMatrix # native

Prints a formatted confusion matrix

function : native : PrintConfusionMatrix(matrix:Int[,]) ~ Nil

Parameters

Recall # native

Calculates recall (sensitivity): TP / (TP + FN)

function : native : Recall(predictions:Bool[], actuals:Bool[]) ~ Float

Parameters

Return

RecallAtFpr # function

The highest recall reachable without exceeding a false-positive rate. This is the number an analyst actually cares about: alerts arrive against a fixed budget, so "how many attacks do we catch while raising no more than one false alarm in a thousand" is the question, not accuracy.

function : RecallAtFpr(scores:Float[], actuals:Bool[], max_fpr:Float) ~ Float

Parameters

Return

Example

r := Metrics->RecallAtFpr(scores, actuals, 0.001);
"recall at 0.1% FPR: {$r}"->PrintLine();

RecallByGroup # function

Recall within each group, at one threshold. An overall recall is an average over whatever mix of sub-populations the sample happens to contain, and a rare one contributes almost nothing to it. A detector can report strong recall while missing a whole category that makes up a fraction of a percent of the data -- which is often the category that matters most, because rarity in a benchmark is not unimportance in a deployment. Recall is computed over the POSITIVES in each group. Rows of a group that are not positive are counted nowhere: there is no recall to measure on them. A group with no positives is still returned, with a count of zero, so a caller iterating the result sees that it was empty rather than finding it missing.

function : RecallByGroup(scores:Float[], actuals:Bool[], groups:Float[], threshold:Float) ~ GroupRecall[]

Parameters

Return

RocCurve # function

The ROC curve as [threshold, false-positive rate, true-positive rate] rows, with the origin included so it can be plotted directly.

function : RocCurve(scores:Float[], actuals:Bool[]) ~ Float[,]

Parameters

Return

ShuffleLabels # function

A seeded permutation of a label array. For a shuffled-label control: a model trained on permuted labels should score near chance, and anything better means information is reaching the model that should not be -- a leak, or a bug in the pipeline rather than in the model. It is the cheapest check there is against a result that is too good. Returns a new array; the input is not modified, because the real labels are almost always still needed afterwards to score the control against something. Seeded so that a control which does find something can be reproduced. An unseeded shuffle that reported an AUC of 0.68 once cannot be investigated.

function : ShuffleLabels(labels:Bool[], seed:Int) ~ Bool[]

Parameters

Return

ThresholdAtFpr # function

The score threshold of the operating point RecallAtFpr selects.

function : ThresholdAtFpr(scores:Float[], actuals:Bool[], max_fpr:Float) ~ Float

Parameters

Return

UnseenGroups # function

Group codes present in one set and absent from another. For reporting on sub-populations a model was never trained on. Recognising patterns resembling the training data is a different task from detecting something genuinely new, and a single overall number cannot tell them apart, so the groups that were never seen have to be identified before they can be reported separately. Order follows first appearance in later, so the result lines up with a RecallByGroup over the same array.

function : UnseenGroups(earlier:Float[], later:Float[]) ~ Float[]

Parameters

Return