Core machine learning types: the seedable Random generator, Matrix2D linear algebra and activations, matrix reference holders, the NeuralNetwork and CSV-backed MatrixReader. Compile with -lib ml.
GradientBoostedClassifier
Gradient boosted trees for binary classification, under logistic loss. GradientBoostedTrees above fits each stage to the plain residual y - F(x), which is the gradient of squared error. For a binary label the right target is the gradient of the logistic loss, y - sigmoid(F(x)). A row the model already calls correctly, and confidently, leaves a residual near zero and stops steering the fit, so later stages spend their capacity on the rows still being missed. That single substitution is the whole difference: the trees, the shrinkage and the loop are the regression version's. PredictScore returns a probability, which is what an evaluation at a fixed false-alarm rate needs; Predict thresholds it half way. Deterministic — the same data gives the same ensemble, with no seed to pass.
Example
objeck
gbc := GradientBoostedClassifier->New(50, 0.1, 3, 5);
gbc->Fit(train_x, train_y);
scores := gbc->PredictScore(test_x);
cut := Metrics->ThresholdAtFpr(test_y, scores, 0.01);Operations
Fit #
Fits the ensemble on binary labels.
method : public : Fit(X:Float[,], y:Bool[]) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features, continuous |
| y | Bool | one label per row, true for the positive class |
Return
| Type | Description |
|---|---|
| Bool | true when every stage was fitted |
GetBase #
The starting score, the log-odds of the training positive rate.
method : public : GetBase() ~ FloatReturn
| Type | Description |
|---|---|
| Float | the base score, or 0.0 when unfitted |
GetTreeCount #
How many boosting stages were fitted.
method : public : GetTreeCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | stage count, or 0 when unfitted |
IsFitted #
Whether Fit has been called successfully.
method : public : IsFitted() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when this ensemble has been fitted |
New # constructor
Constructor
New(num_trees:Int, learning_rate:Float, max_depth:Int, min_samples:Int)Parameters
| Name | Type | Description |
|---|---|---|
| num_trees | Int | boosting stages |
| learning_rate | Float | shrinkage applied to each stage, e.g. 0.1 |
| max_depth | Int | deepest any stage's tree may grow |
| min_samples | Int | a node with this many rows or fewer becomes a leaf |
Predict #
Labels at the usual half-way threshold. Prefer PredictScore with an explicit threshold when the false-alarm rate matters. Tested against the raw score, since sigmoid(0) is one half and the two agree exactly.
method : public : Predict(X:Float[,]) ~ Bool[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
Return
| Type | Description |
|---|---|
| Bool | one label per row, or Nil when unfitted |
PredictRaw #
The ensemble's raw score for each row: the log-odds, before the sigmoid. Ranking rows by this gives the same order as ranking by the probability, since the sigmoid is monotone, and skips a pass over the data.
method : public : PredictRaw(X:Float[,]) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
Return
| Type | Description |
|---|---|
| Float | one score per row, or Nil when unfitted |