Experiment support for a parameter sweep: a deterministic stratified hold-out split, class-ratio resampling of the training half against a fixed test set, a wall-clock stopwatch and label helpers. Compile with -lib ml,ml_lab,csv,json, or with the ai alias group. Design: docs/ML_LAB_SUPPORT_DESIGN.md.
Experiment
A sweep over one named parameter: for every value (and every seed, when repeats are asked for) it runs the Trial, scores the predictions, and collects a ResultTable with train_accuracy, test_accuracy, precision, recall, f1, fpr, false_negatives and fit_ms per value -- mean and sample standard deviation over the seeds when there is more than one.
Test-set metrics come from the test predictions; train_accuracy is the only training-set number, there for the overfitting curve. The confusion counts and rates come from System.ML's ConfusionMatrix over the trial's Bool predictions, every ratio guarded (a value that predicts nothing positive has no precision and reports 0.0).
Example
exp := Experiment->New("Random forest: trees", "n_estimators");
exp->SetValues([10.0, 50.0, 100.0, 200.0, 500.0]);
exp->SetSeeds([1, 2, 3]);
exp->AddSetting("max_depth", "8");
table := exp->Run(ForestTrial->New(split));
table->ToMarkdown()->PrintLine();Operations
AddSetting #
Records a fixed setting for the footnote, e.g. ("max_depth", "8").
method : public : AddSetting(name:String, value:String) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| name | String | the setting |
| value | String | its value |
New # constructor
Constructor
New(title:String, parameter:String)Parameters
| Name | Type | Description |
|---|---|---|
| title | String | the table's title |
| parameter | String | the swept parameter's name |
Run #
Runs the sweep.
method : public : Run(trial:Trial) ~ ResultTableParameters
| Name | Type | Description |
|---|---|---|
| trial | Trial | builds, fits and predicts for one value |
Return
| Type | Description |
|---|---|
| ResultTable | the table, one row per value; raises LabError when no values are set, the trial returns Nil, or its arrays disagree in length |
SetSeeds #
Seeds for repeated runs; each value is run once per seed and the table reports the mean and sample standard deviation. The default is one run with seed 0.
method : public : SetSeeds(seeds:Int[]) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| seeds | Int | one per repetition |
SetValues #
The values to sweep, named by their number.
method : public : SetValues(values:Float[]) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| values | Float | one per trial |
SetValues #
The values to sweep with display names, for a categorical parameter such as a kernel (values 0, 1, 2; names "linear", "rbf", "poly").
method : public : SetValues(values:Float[], names:String[]) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| values | Float | one per trial |
| names | String | one per value, or Nil to name each by its number |