What a student needs to turn a study into a hand-in: Replicates collects the value of each metric per seed and per setting and reports the count, mean, sample standard deviation, range and a t confidence interval, with a paired comparison of two settings; ResultTables converts between a System.ML.Lab ResultTable and the other results (replicates, an OLS fit, t-tests, a grid of strings); Report builds one document with numbered sections, tables, figures, code and notes, written as Markdown or as one self-contained HTML file with a print stylesheet that prints cleanly to PDF from a browser.
No call here ends the program on bad input: a call that cannot do what it was asked returns false (or Nil) and the object keeps the reason, read with GetError(), which is "" when there is nothing to report. Output is deterministic: the same calls give the same bytes, and nothing reads a clock. Compile with -lib ml_report,ml_lab,ml,stats,plot,csv,json,gen_collect.
Replicates
Per-seed values of named metrics under named settings (groups), with the summary a hand-in reports. Every statistic leaves out a value that is not finite (NaN, an infinity, or the value given to SetMissing) and says how many it left out; a statistic that needs more values than there are is NaN, never an error, so a one-seed run gives a mean and an undefined standard deviation.
Example
rep := Replicates->New();
each(seed : 5) {
rep->Add("depth 4", "test_accuracy", seed, 0.80 + seed * 0.01);
};
rep->GetMean("depth 4", "test_accuracy")->PrintLine();
rep->GetCiHigh("depth 4", "test_accuracy")->PrintLine();Operations
- New
- Add
- AddRun
- GetCiHigh
- GetCiLow
- GetConfidence
- GetCount
- GetError
- GetGroups
- GetMax
- GetMean
- GetMetrics
- GetMin
- GetMissing
- GetPairedCount
- GetSeeds
- GetStd
- GetStdError
- GetUsable
- GetValues
- Paired
- SetConfidence
- SetMissing
- ToResultTable
- Welch
Add #
Records one value.
method : public : Add(group:String, metric:String, seed:Int, value:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting, for example "depth 4" or "baseline" |
| metric | String | the metric, for example "test_accuracy" |
| seed | Int | the seed, so that two settings can be paired by seed |
| value | Float | the value; NaN and infinities are kept and counted as missing |
Return
| Type | Description |
|---|---|
| Bool | false, with GetError() saying why, for a Nil or empty name or a seed already recorded for that setting and metric |
Add #
Records one value under the next free seed number of that setting and metric (0, 1, 2, ...).
method : public : Add(group:String, metric:String, value:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
| value | Float | the value |
Return
| Type | Description |
|---|---|
| Bool | false when a name is Nil or empty |
AddRun #
Records one run: every metric of one seed under one setting.
method : public : AddRun(group:String, seed:Int, metrics:String[], values:Float[]) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| seed | Int | the seed |
| metrics | String | the metric names |
| values | Float | one value per metric |
Return
| Type | Description |
|---|---|
| Bool | false when the arrays differ in length or any value is refused; the values before a refused one stay recorded |
GetCiHigh #
The upper end of the t confidence interval of the mean.
method : public : GetCiHigh(group:String, metric:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the bound, NaN with fewer than 2 usable values |
GetCiLow #
The lower end of the t confidence interval of the mean (mean - t * s / sqrt(n), n - 1 degrees of freedom).
method : public : GetCiLow(group:String, metric:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the bound, NaN with fewer than 2 usable values |
GetConfidence #
The confidence level.
method : public : GetConfidence() ~ FloatReturn
| Type | Description |
|---|---|
| Float | the level |
GetCount #
How many values were usable (finite and not the missing marker).
method : public : GetCount(group:String, metric:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Int | the count, 0 for an unknown setting or metric |
GetError #
Why the last call that returns Bool was refused.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, "" when it succeeded |
GetGroups #
The setting names in the order they were first recorded.
method : public : GetGroups() ~ String[]Return
| Type | Description |
|---|---|
| String | the names |
GetMax #
The largest usable value.
method : public : GetMax(group:String, metric:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the maximum, NaN when none is usable |
GetMean #
The mean of the usable values.
method : public : GetMean(group:String, metric:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the mean, NaN when none is usable |
GetMetrics #
The metric names in the order they were first recorded.
method : public : GetMetrics() ~ String[]Return
| Type | Description |
|---|---|
| String | the names |
GetMin #
The smallest usable value.
method : public : GetMin(group:String, metric:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the minimum, NaN when none is usable |
GetMissing #
How many recorded values were left out as missing.
method : public : GetMissing(group:String, metric:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Int | the count |
GetPairedCount #
How many seeds two settings share with a usable value in both: the n of the paired comparison.
method : public : GetPairedCount(a:String, b:String, metric:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| a | String | the first setting |
| b | String | the second setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Int | the count |
GetSeeds #
The seeds of GetValues, in the same order.
method : public : GetSeeds(group:String, metric:String) ~ Int[]Parameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Int | the seeds, empty when there are none |
GetStd #
The sample standard deviation (divisor n - 1, two-pass) of the usable values.
method : public : GetStd(group:String, metric:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the deviation, NaN with fewer than 2 usable values |
GetStdError #
The standard error of the mean: the standard deviation over the square root of the count.
method : public : GetStdError(group:String, metric:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the standard error, NaN with fewer than 2 usable values |
GetUsable #
The values that count: the recorded ones without the missing.
method : public : GetUsable(group:String, metric:String) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the usable values in the order added |
GetValues #
Every value recorded, non-finite ones included, in the order added.
method : public : GetValues(group:String, metric:String) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| group | String | the setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| Float | the values, empty when there are none |
Paired #
Paired t-test of two settings over the seeds they share (System.Stats TTest->Paired): the mean of a minus b, its t statistic, degrees of freedom, two-sided p-value, confidence interval and effect sizes.
method : public : Paired(a:String, b:String, metric:String) ~ TTestResultParameters
| Name | Type | Description |
|---|---|---|
| a | String | the first setting |
| b | String | the second setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| TTestResult | the result; IsValid() is false, with the reason in GetError(), with fewer than 2 shared seeds |
SetConfidence #
Sets the confidence level of the interval of the mean and of the paired comparison.
method : public : SetConfidence(level:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| level | Float | strictly between 0 and 1 |
Return
| Type | Description |
|---|---|
| Bool | false, with GetError() saying why, when the level is not in (0, 1) |
SetMissing #
Treats one exact value as "no result", for a metric that reports a sentinel such as the -1.0 of Metrics->AucRoc when a class is missing. The value is kept, counted by GetMissing and left out of the statistics.
method : public : SetMissing(sentinel:Float) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| sentinel | Float | the marker value |
ToResultTable #
One metric as a ResultTable with a row per setting and the columns mean, std, ci_low, ci_high, n and missing. See ResultTables->FromReplicates.
method : public : ToResultTable(metric:String, title:String, parameter:String) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| metric | String | the metric |
| title | String | the table title |
| parameter | String | the heading of the first column |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil when the metric is unknown |
Welch #
Welch's unequal-variance t-test of two settings, which do not need to share seeds or counts.
method : public : Welch(a:String, b:String, metric:String) ~ TTestResultParameters
| Name | Type | Description |
|---|---|---|
| a | String | the first setting |
| b | String | the second setting |
| metric | String | the metric |
Return
| Type | Description |
|---|---|
| TTestResult | the result; IsValid() is false with fewer than 2 usable values in either |