v2026.11.2
All Bundles
Bundle

What a student needs to turn a study into a hand-in: Replicates collects the value of each metric per seed and per setting and reports the count, mean, sample standard deviation, range and a t confidence interval, with a paired comparison of two settings; ResultTables converts between a System.ML.Lab ResultTable and the other results (replicates, an OLS fit, t-tests, a grid of strings); Report builds one document with numbered sections, tables, figures, code and notes, written as Markdown or as one self-contained HTML file with a print stylesheet that prints cleanly to PDF from a browser.

No call here ends the program on bad input: a call that cannot do what it was asked returns false (or Nil) and the object keeps the reason, read with GetError(), which is "" when there is nothing to report. Output is deterministic: the same calls give the same bytes, and nothing reads a clock. Compile with -lib ml_report,ml_lab,ml,stats,plot,csv,json,gen_collect.

Replicates

Per-seed values of named metrics under named settings (groups), with the summary a hand-in reports. Every statistic leaves out a value that is not finite (NaN, an infinity, or the value given to SetMissing) and says how many it left out; a statistic that needs more values than there are is NaN, never an error, so a one-seed run gives a mean and an undefined standard deviation.

Example

rep := Replicates->New();
each(seed : 5) {
  rep->Add("depth 4", "test_accuracy", seed, 0.80 + seed * 0.01);
};
rep->GetMean("depth 4", "test_accuracy")->PrintLine();
rep->GetCiHigh("depth 4", "test_accuracy")->PrintLine();

Operations

Add #

Records one value.

method : public : Add(group:String, metric:String, seed:Int, value:Float) ~ Bool

Parameters

NameTypeDescription
groupStringthe setting, for example "depth 4" or "baseline"
metricStringthe metric, for example "test_accuracy"
seedIntthe seed, so that two settings can be paired by seed
valueFloatthe value; NaN and infinities are kept and counted as missing

Return

TypeDescription
Boolfalse, with GetError() saying why, for a Nil or empty name or a seed already recorded for that setting and metric

Add #

Records one value under the next free seed number of that setting and metric (0, 1, 2, ...).

method : public : Add(group:String, metric:String, value:Float) ~ Bool

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric
valueFloatthe value

Return

TypeDescription
Boolfalse when a name is Nil or empty

AddRun #

Records one run: every metric of one seed under one setting.

method : public : AddRun(group:String, seed:Int, metrics:String[], values:Float[]) ~ Bool

Parameters

NameTypeDescription
groupStringthe setting
seedIntthe seed
metricsStringthe metric names
valuesFloatone value per metric

Return

TypeDescription
Boolfalse when the arrays differ in length or any value is refused; the values before a refused one stay recorded

GetCiHigh #

The upper end of the t confidence interval of the mean.

method : public : GetCiHigh(group:String, metric:String) ~ Float

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe bound, NaN with fewer than 2 usable values

GetCiLow #

The lower end of the t confidence interval of the mean (mean - t * s / sqrt(n), n - 1 degrees of freedom).

method : public : GetCiLow(group:String, metric:String) ~ Float

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe bound, NaN with fewer than 2 usable values

GetConfidence #

The confidence level.

method : public : GetConfidence() ~ Float

Return

TypeDescription
Floatthe level

GetCount #

How many values were usable (finite and not the missing marker).

method : public : GetCount(group:String, metric:String) ~ Int

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Intthe count, 0 for an unknown setting or metric

GetError #

Why the last call that returns Bool was refused.

method : public : GetError() ~ String

Return

TypeDescription
Stringthe reason, "" when it succeeded

GetGroups #

The setting names in the order they were first recorded.

method : public : GetGroups() ~ String[]

Return

TypeDescription
Stringthe names

GetMax #

The largest usable value.

method : public : GetMax(group:String, metric:String) ~ Float

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe maximum, NaN when none is usable

GetMean #

The mean of the usable values.

method : public : GetMean(group:String, metric:String) ~ Float

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe mean, NaN when none is usable

GetMetrics #

The metric names in the order they were first recorded.

method : public : GetMetrics() ~ String[]

Return

TypeDescription
Stringthe names

GetMin #

The smallest usable value.

method : public : GetMin(group:String, metric:String) ~ Float

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe minimum, NaN when none is usable

GetMissing #

How many recorded values were left out as missing.

method : public : GetMissing(group:String, metric:String) ~ Int

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Intthe count

GetPairedCount #

How many seeds two settings share with a usable value in both: the n of the paired comparison.

method : public : GetPairedCount(a:String, b:String, metric:String) ~ Int

Parameters

NameTypeDescription
aStringthe first setting
bStringthe second setting
metricStringthe metric

Return

TypeDescription
Intthe count

GetSeeds #

The seeds of GetValues, in the same order.

method : public : GetSeeds(group:String, metric:String) ~ Int[]

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Intthe seeds, empty when there are none

GetStd #

The sample standard deviation (divisor n - 1, two-pass) of the usable values.

method : public : GetStd(group:String, metric:String) ~ Float

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe deviation, NaN with fewer than 2 usable values

GetStdError #

The standard error of the mean: the standard deviation over the square root of the count.

method : public : GetStdError(group:String, metric:String) ~ Float

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe standard error, NaN with fewer than 2 usable values

GetUsable #

The values that count: the recorded ones without the missing.

method : public : GetUsable(group:String, metric:String) ~ Float[]

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe usable values in the order added

GetValues #

Every value recorded, non-finite ones included, in the order added.

method : public : GetValues(group:String, metric:String) ~ Float[]

Parameters

NameTypeDescription
groupStringthe setting
metricStringthe metric

Return

TypeDescription
Floatthe values, empty when there are none

New # constructor

Constructor: a 95% confidence level and no missing-value marker.

New()

Paired #

Paired t-test of two settings over the seeds they share (System.Stats TTest->Paired): the mean of a minus b, its t statistic, degrees of freedom, two-sided p-value, confidence interval and effect sizes.

method : public : Paired(a:String, b:String, metric:String) ~ TTestResult

Parameters

NameTypeDescription
aStringthe first setting
bStringthe second setting
metricStringthe metric

Return

TypeDescription
TTestResultthe result; IsValid() is false, with the reason in GetError(), with fewer than 2 shared seeds

SetConfidence #

Sets the confidence level of the interval of the mean and of the paired comparison.

method : public : SetConfidence(level:Float) ~ Bool

Parameters

NameTypeDescription
levelFloatstrictly between 0 and 1

Return

TypeDescription
Boolfalse, with GetError() saying why, when the level is not in (0, 1)

SetMissing #

Treats one exact value as "no result", for a metric that reports a sentinel such as the -1.0 of Metrics->AucRoc when a class is missing. The value is kept, counted by GetMissing and left out of the statistics.

method : public : SetMissing(sentinel:Float) ~ Nil

Parameters

NameTypeDescription
sentinelFloatthe marker value

ToResultTable #

One metric as a ResultTable with a row per setting and the columns mean, std, ci_low, ci_high, n and missing. See ResultTables->FromReplicates.

method : public : ToResultTable(metric:String, title:String, parameter:String) ~ ResultTable

Parameters

NameTypeDescription
metricStringthe metric
titleStringthe table title
parameterStringthe heading of the first column

Return

TypeDescription
ResultTablethe table, Nil when the metric is unknown

Welch #

Welch's unequal-variance t-test of two settings, which do not need to share seeds or counts.

method : public : Welch(a:String, b:String, metric:String) ~ TTestResult

Parameters

NameTypeDescription
aStringthe first setting
bStringthe second setting
metricStringthe metric

Return

TypeDescription
TTestResultthe result; IsValid() is false with fewer than 2 usable values in either