Core machine learning types: the seedable Random generator, Matrix2D linear algebra and activations, matrix reference holders, the NeuralNetwork and CSV-backed MatrixReader. Compile with -lib ml.
EmbeddingClassifier
A binary classifier trained on fixed-size feature vectors -- sentence embeddings from a frozen encoder above all: logistic regression (no hidden layer) or a small multi-layer network with ReLU hidden layers, a sigmoid output and binary cross-entropy loss, trained by minibatch Adam or SGD at a configurable learning rate.
For class imbalance, SetClassWeights (or SetBalancedWeights) weights each row's loss by its class. GetLossHistory gives the training loss after every epoch, for a loss curve.
Inputs are standardised with the training set's mean and deviation, which are kept with the model: embedding dimensions differ widely in scale, and an unscaled head trains far slower. Weights start from System.ML.Random with the given seed, and minibatches are shuffled by the same generator: the same seed and data give the same model, bit for bit.
This is the head of a frozen-encoder classifier (docs/ML_BERT_DESIGN.md); nothing here trains the encoder.
Example
head := EmbeddingClassifier->New([64], 0.001, 20, 32, 42);
head->SetBalancedWeights();
head->Fit(X_train, y_train);
loss := head->GetLossHistory(); # one value per epoch
p := head->PredictProba(X_test); # P(positive) per row
yhat := head->PredictClass(X_test, 0.5);Operations
- New
- Fit
- GetBias
- GetClassWeights
- GetError
- GetLossHistory
- GetParameterCount
- GetWeights
- IsFitted
- Loss
- PredictClass
- PredictProba
- PredictScore
- SetBalancedWeights
- SetClassWeights
- SetL2
- SetOptimizer
- SetStandardize
Fit #
Trains the head.
method : public : Fit(X:Float[,], y:Bool[]) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
| y | Bool | one label per row, true for the positive class |
Return
| Type | Description |
|---|---|
| Bool | true when trained; false with GetError set for empty, mismatched or NaN input, or a configuration that cannot train |
GetBias #
A layer's biases, for inspection. A copy.
method : public : GetBias(layer:Int) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| layer | Int | layer index |
Return
| Type | Description |
|---|---|
| Float | biases, or Nil before Fit or for a layer out of range |
GetClassWeights #
The class weights in use: those set, or after a balanced Fit, those computed.
method : public : GetClassWeights() ~ Float[]Return
| Type | Description |
|---|---|
| Float | [negative, positive] |
GetError #
Why the last Fit returned false or the last prediction Nil.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or Nil |
GetLossHistory #
The training loss after each epoch: the class-weighted binary cross-entropy over the whole training set, averaged over its rows, with L2 left out.
method : public : GetLossHistory() ~ Float[]Return
| Type | Description |
|---|---|
| Float | one value per epoch, or Nil before Fit |
GetParameterCount #
Number of trainable parameters, weights and biases.
method : public : GetParameterCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | parameter count, 0 before Fit |
GetWeights #
A layer's weights, inputs by outputs, for inspection: layer 0 takes the (standardised) input, the last gives the logit. A copy.
method : public : GetWeights(layer:Int) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| layer | Int | layer index |
Return
| Type | Description |
|---|---|
| Float | weights, or Nil before Fit or for a layer out of range |
IsFitted #
Whether Fit has succeeded.
method : public : IsFitted() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when the model can predict |
Loss #
The class-weighted binary cross-entropy of the model on labelled rows, averaged over the rows: the measure GetLossHistory records, for a validation curve.
method : public : Loss(X:Float[,], y:Bool[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
| y | Bool | one label per row |
Return
| Type | Description |
|---|---|
| Float | the loss, or -1.0 when X or y cannot be scored (see GetError) |
New # constructor
A head with one hidden layer of 64, learning rate 0.001, 20 epochs, batch 32, seed 42.
New()New # constructor
A head.
New(hidden:Int[], learning_rate:Float, epochs:Int, batch_size:Int, seed:Int)Parameters
| Name | Type | Description |
|---|---|---|
| hidden | Int | hidden-layer sizes; an empty array is logistic regression |
| learning_rate | Float | step size |
| epochs | Int | passes over the training data |
| batch_size | Int | rows per update |
| seed | Int | seed for the initial weights and the shuffles |
PredictClass #
Predicts each row's class: positive when its probability is at least threshold.
method : public : PredictClass(X:Float[,], threshold:Float) ~ Bool[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
| threshold | Float | the cut, e.g. 0.5 |
Return
| Type | Description |
|---|---|
| Bool | one label per row, or Nil (see GetError) |
PredictClass #
Predicts each row's class at a threshold of 0.5.
method : public : PredictClass(X:Float[,]) ~ Bool[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
Return
| Type | Description |
|---|---|
| Bool | one label per row, or Nil (see GetError) |
PredictProba #
The probability of the positive class for each row.
method : public : PredictProba(X:Float[,]) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features, as many as Fit saw |
Return
| Type | Description |
|---|---|
| Float | one probability per row, or Nil (see GetError) |
PredictScore #
Scores each row, higher meaning more positive: the positive-class probability, for CrossValidation and the threshold tools in Metrics.
method : public : PredictScore(X:Float[,]) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
Return
| Type | Description |
|---|---|
| Float | one score per row, or Nil (see GetError) |
SetBalancedWeights #
Weights each class by n / (2 x its count) at Fit, so both classes carry equal total weight -- scikit-learn's class_weight="balanced".
method : public : SetBalancedWeights() ~ NilSetClassWeights #
Weights each training row's loss by its class, for imbalanced data: a positive weight of 99 makes one missed positive cost as much as 99 false alarms. Both 1.0 by default.
method : public : SetClassWeights(negative:Float, positive:Float) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| negative | Float | weight of a negative row |
| positive | Float | weight of a positive row |
SetL2 #
Sets L2 regularisation of the weights (not the biases); 0.0 by default.
method : public : SetL2(l2:Float) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| l2 | Float | strength |
SetOptimizer #
Sets the optimizer; Adam by default.
method : public : SetOptimizer(optimizer:EmbeddingClassifier->Optimizer) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| optimizer | EmbeddingClassifier->Optimizer | ADAM or SGD |