What a student needs to turn a study into a hand-in: Replicates collects the value of each metric per seed and per setting and reports the count, mean, sample standard deviation, range and a t confidence interval, with a paired comparison of two settings; ResultTables converts between a System.ML.Lab ResultTable and the other results (replicates, an OLS fit, t-tests, a grid of strings); Report builds one document with numbered sections, tables, figures, code and notes, written as Markdown or as one self-contained HTML file with a print stylesheet that prints cleanly to PDF from a browser.
No call here ends the program on bad input: a call that cannot do what it was asked returns false (or Nil) and the object keeps the reason, read with GetError(), which is "" when there is nothing to report. Output is deterministic: the same calls give the same bytes, and nothing reads a clock. Compile with -lib ml_report,ml_lab,ml,stats,plot,csv,json,gen_collect.
ResultTables
Conversions between System.ML.Lab's ResultTable and the rest: summaries of replicates, an OLS fit, a list of t-tests, and a grid of strings. A conversion that cannot be done returns Nil and GetError() says why ("" when the last one worked).
Operations
- CsvColumnKinds
- CsvToCells
- FromCells
- FromComparison
- FromCsv
- FromOls
- FromReplicates
- FromTTests
- GetError
- MeansFromReplicates
- ToCells
CsvColumnKinds # function
The kind of each column of a grid (row 0 is the heading): "integer" when every non-empty cell is a whole number, "number" when every one is a number, "empty" when there is none, else "text". A cell that is empty or "n/a" is skipped.
function : CsvColumnKinds(cells:String[,]) ~ String[]Parameters
| Name | Type | Description |
|---|---|---|
| cells | String | the grid, for example from CsvToCells |
Return
| Type | Description |
|---|---|
| String | one kind per column, Nil for a Nil or empty grid |
CsvToCells # function
Any CSV text as a grid of text for Report->AddTable: row 0 is the heading, each further row is a record. Use it for a CSV that System.ML.Lab did not write.
function : CsvToCells(text:String) ~ String[,]Parameters
| Name | Type | Description |
|---|---|---|
| text | String | CSV text with a heading row and at least one record |
Return
| Type | Description |
|---|---|
| String | the grid, Nil with the reason in GetError() for Nil text or CSV that does not parse (an empty text, no records, a record with the wrong number of fields) |
FromCells # function
A grid of text as a ResultTable: row 0 names the parameter (cell 0) and the columns, each further row is a name then one cell per column. A cell is read as a number; "mean ± std" gives the cell a deviation; anything else (n/a, empty, words) is NaN. Column names must be unique and not empty.
function : FromCells(title:String, cells:String[,]) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| title | String | the title |
| cells | String | the grid, at least a heading row and one column |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil with the reason in GetError() when the grid cannot be a table |
FromComparison # function
A paired comparison of two settings across all metrics: a row per metric with the two means, their difference, its confidence interval, t, df, p_value, cohen_d and n_pairs. A metric with fewer than 2 shared seeds has NaN in the test columns.
function : FromComparison(rep:Replicates, a:String, b:String, title:String) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| rep | Replicates | the replicates |
| a | String | the first setting |
| b | String | the second setting |
| title | String | the title |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil when rep is Nil or a setting is unknown |
FromCsv # function
A System.ML.Lab ResultTable from CSV text written by ResultTable->ToCsv, without ending the program: ResultTable->FromCsv raises a LabError for text that is not in that layout, this returns Nil and GetError() says why. The layout is a heading of the parameter name, "
function : FromCsv(text:String) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| text | String | CSV text |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil with the reason in GetError() for Nil or empty text, CSV that does not parse, or a heading that is not that layout |
FromOls # function
An OLS fit as a table: a row per coefficient with the columns coef, std_err, t, p_value, ci_low and ci_high (95%); the footnote carries n, R-squared and the F test.
function : FromOls(fit:OLSResult, title:String) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| fit | OLSResult | the fit from System.Stats OLS->Fit |
| title | String | the title |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil when the fit is Nil or not valid |
FromReplicates # function
One metric of a Replicates as a table: a row per setting, the columns mean, std, ci_low, ci_high, n (usable values) and missing. A row whose setting name is a number carries it as the row value (so a chart can use it for x), otherwise the row number.
function : FromReplicates(rep:Replicates, metric:String, title:String, parameter:String) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| rep | Replicates | the replicates |
| metric | String | the metric |
| title | String | the title |
| parameter | String | the heading of the first column, for example "max_depth" |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil when rep is Nil or the metric was never recorded |
FromTTests # function
A list of t-tests as a table: a row per test with mean_diff, t, df, p_value, ci_low, ci_high, cohen_d, hedges_g, n1 and n2. An invalid test is a row of NaN, and the footnote says why.
function : FromTTests(names:String[], tests:TTestResult[], title:String) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| names | String | one name per test |
| tests | TTestResult | the results from System.Stats TTest |
| title | String | the title |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil when the arrays are Nil or differ in length |
GetError # function
Why the last conversion returned Nil.
function : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, "" when it worked |
MeansFromReplicates # function
Every metric of a Replicates as one table in the sweep layout: a row per setting, a column per metric holding the mean, and the sample standard deviation as the cell's deviation, which renders as "mean ± std". A one-seed cell has an undefined deviation.
function : MeansFromReplicates(rep:Replicates, title:String, parameter:String) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| rep | Replicates | the replicates |
| title | String | the title |
| parameter | String | the heading of the first column |
Return
| Type | Description |
|---|---|
| ResultTable | the table, Nil when rep is Nil or empty |
ToCells # function
A ResultTable as a grid of text: row 0 is the heading (the parameter name, then the columns) and each further row is a row name then its cells, spelled as ResultTable->ToMarkdown() spells them (so the column's decimals and "mean ± std" are kept, and NaN reads n/a).
function : ToCells(table:ResultTable) ~ String[,]Parameters
| Name | Type | Description |
|---|---|---|
| table | ResultTable | the table |
Return
| Type | Description |
|---|---|
| String | the grid, Nil when the table is Nil |