Core machine learning types: the seedable Random generator, Matrix2D linear algebra and activations, matrix reference holders, the NeuralNetwork and CSV-backed MatrixReader. Compile with -lib ml.
KernelSVM
A kernel support vector machine for binary classification (C-SVC), with scikit-learn's SVC parameters: kernel, C, gamma ('scale', 'auto' or a value), degree, coef0, tol, shrinking, cache size, max_iter and class weights.
It is trained by LIBSVM's SMO solver -- second-order working-set selection, shrinking, and an LRU cache of kernel rows sized in megabytes -- so it finds the exact dual optimum to the tolerance, deterministically: there is no randomness in fitting and no seed. The linear SVM class is a different model (a primal subgradient method parameterised by lambda); use this one for kernels or for scikit-learn's C.
Cost grows between n^2 and n^3 in the training rows, as for LIBSVM: tens of thousands of rows is practical, the full 126k-row NSL-KDD training set is not. Subsample for parameter sweeps. Scale the features first (FeatureScaler): RBF, polynomial and sigmoid kernels are distances and dot products, and an unscaled column dominates them.
DecisionFunction returns a signed margin, higher meaning more positive, which is the score Metrics->AucRoc, RecallAtFpr and ThresholdAtFpr take.
Example
svm := KernelSVM->New(SvmKernel->RBF, 10.0, 0.1);
if(svm->Fit(x_train, y_train)) {
scores := svm->DecisionFunction(x_test);
auc := Metrics->AucRoc(scores, y_test);
labels := svm->PredictClass(x_test);
};Operations
- New
- DecisionFunction
- Fit
- GetBoundedSupportVectorCount
- GetC
- GetCacheSize
- GetCoef0
- GetDegree
- GetDualCoefficients
- GetFitSeconds
- GetGamma
- GetIntercept
- GetIterations
- GetKernel
- GetLastError
- GetMaxIterations
- GetObjective
- GetSupportIndices
- GetSupportVectorCount
- GetSupportVectors
- GetTolerance
- IsConverged
- IsFitted
- IsShrinking
- PredictClass
- PredictScore
- Score
- SetC
- SetCacheSize
- SetClassWeight
- SetClassWeightBalanced
- SetCoef0
- SetDegree
- SetGamma
- SetGammaAuto
- SetGammaScale
- SetMaxIterations
- SetShrinking
- SetTolerance
DecisionFunction #
The signed distance-like margin of each row: sum of alpha_i y_i K(x_i, x) over the support vectors, plus the intercept. Positive means the true class; the magnitude grows with confidence but is not a probability. This is the score for Metrics->AucRoc, RecallAtFpr and ThresholdAtFpr.
method : public : DecisionFunction(X:Float[,]) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features, with the column count used at Fit |
Return
| Type | Description |
|---|---|
| Float | one value per row, or Nil when unfitted or the shape differs |
Fit #
Fits the model.
method : public : Fit(X:Float[,], y:Bool[]) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
| y | Bool | one label per row, true for the positive class |
Return
| Type | Description |
|---|---|
| Bool | true when fitted; false on bad input or parameters, with GetLastError saying why |
GetBoundedSupportVectorCount #
Number of support vectors at their upper bound C (margin violators).
method : public : GetBoundedSupportVectorCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | count, or 0 when unfitted |
GetCacheSize #
The kernel-row cache size.
method : public : GetCacheSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | megabytes |
GetCoef0 #
The POLY/SIGMOID independent term.
method : public : GetCoef0() ~ FloatReturn
| Type | Description |
|---|---|
| Float | coef0 |
GetDualCoefficients #
alpha_i * y_i per support vector (y as +1/-1), scikit-learn's dual_coef_.
method : public : GetDualCoefficients() ~ Float[]Return
| Type | Description |
|---|---|
| Float | coefficients, or Nil when unfitted |
GetFitSeconds #
Seconds the last Fit took.
method : public : GetFitSeconds() ~ FloatReturn
| Type | Description |
|---|---|
| Float | elapsed seconds |
GetGamma #
The gamma in use: after Fit the value the kernel used, including the one 'scale' or 'auto' resolved to; before Fit the explicit value, or -1.0 for 'scale' and -2.0 for 'auto'.
method : public : GetGamma() ~ FloatReturn
| Type | Description |
|---|---|
| Float | gamma |
GetIntercept #
The intercept b of the decision function (-rho in LIBSVM's terms).
method : public : GetIntercept() ~ FloatReturn
| Type | Description |
|---|---|
| Float | intercept |
GetIterations #
Solver iterations used by the last Fit.
method : public : GetIterations() ~ IntReturn
| Type | Description |
|---|---|
| Int | iterations |
GetLastError #
Why the last Fit returned false.
method : public : GetLastError() ~ StringReturn
| Type | Description |
|---|---|
| String | a reason, or an empty string after a successful Fit |
GetMaxIterations #
The iteration cap as set; 0 or less means LIBSVM's own.
method : public : GetMaxIterations() ~ IntReturn
| Type | Description |
|---|---|
| Int | cap |
GetObjective #
The dual objective 1/2 a'Qa - sum(a) at the solution.
method : public : GetObjective() ~ FloatReturn
| Type | Description |
|---|---|
| Float | objective |
GetSupportIndices #
Training-row indexes of the support vectors, ascending (scikit-learn's support_, which groups them by class instead).
method : public : GetSupportIndices() ~ Int[]Return
| Type | Description |
|---|---|
| Int | indexes, or Nil when unfitted |
GetSupportVectorCount #
Number of support vectors (training rows with a non-zero dual coefficient).
method : public : GetSupportVectorCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | count, or 0 when unfitted |
GetSupportVectors #
The support vectors, one row each, in GetSupportIndices order.
method : public : GetSupportVectors() ~ Float[,]Return
| Type | Description |
|---|---|
| Float | support vectors x features, or Nil when unfitted |
GetTolerance #
The stopping tolerance.
method : public : GetTolerance() ~ FloatReturn
| Type | Description |
|---|---|
| Float | tol |
IsConverged #
Whether the last Fit met the tolerance; false when the iteration cap stopped it first.
method : public : IsConverged() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when converged |
IsFitted #
Whether Fit has succeeded.
method : public : IsFitted() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when fitted |
IsShrinking #
Whether shrinking is on.
method : public : IsShrinking() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when shrinking |
New # constructor
Constructor with scikit-learn's defaults: C = 1.0, gamma = 'scale', degree = 3, coef0 = 0.0, tol = 0.001, shrinking on, a 200 MB cache and no iteration cap beyond LIBSVM's own.
New(kernel:SvmKernel)Parameters
| Name | Type | Description |
|---|---|---|
| kernel | SvmKernel | kernel function |
New # constructor
Constructor with a regularisation strength; gamma is 'scale'.
New(kernel:SvmKernel, c:Float)Parameters
| Name | Type | Description |
|---|---|---|
| kernel | SvmKernel | kernel function |
| c | Float | C, the penalty on margin violations; must be > 0 |
New # constructor
Constructor with C and an explicit gamma.
New(kernel:SvmKernel, c:Float, gamma:Float)Parameters
| Name | Type | Description |
|---|---|---|
| kernel | SvmKernel | kernel function |
| c | Float | C, the penalty on margin violations; must be > 0 |
| gamma | Float | kernel coefficient for RBF, POLY and SIGMOID; must be >= 0 |
New # constructor
Constructor with every kernel parameter.
New(kernel:SvmKernel, c:Float, gamma:Float, degree:Int, coef0:Float)Parameters
| Name | Type | Description |
|---|---|---|
| kernel | SvmKernel | kernel function |
| c | Float | C, the penalty on margin violations; must be > 0 |
| gamma | Float | kernel coefficient for RBF, POLY and SIGMOID; must be >= 0 |
| degree | Int | POLY degree; must be >= 0 |
| coef0 | Float | independent term of POLY and SIGMOID |
PredictClass #
Class labels: true where the decision value is above zero. Prefer DecisionFunction with Metrics->ThresholdAtFpr when false alarms are budgeted.
method : public : PredictClass(X:Float[,]) ~ Bool[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
Return
| Type | Description |
|---|---|
| Bool | one label per row, or Nil when unfitted |
PredictScore #
The ScoreModel score: the same values as DecisionFunction.
method : public : PredictScore(X:Float[,]) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
Return
| Type | Description |
|---|---|
| Float | one score per row, or Nil when unfitted |
Score #
Accuracy of PredictClass against labels.
method : public : Score(X:Float[,], y:Bool[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| X | Float | rows x features |
| y | Bool | one label per row |
Return
| Type | Description |
|---|---|
| Float | fraction correct, or 0.0 when unfitted or the sizes differ |
SetC #
Sets C, the penalty on margin violations.
method : public : SetC(c:Float) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| c | Float | C; must be > 0 |
SetCacheSize #
Sets the kernel-row cache size. Larger caches recompute fewer kernel rows; the result does not depend on it.
method : public : SetCacheSize(megabytes:Int) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| megabytes | Int | cache size in MB; must be >= 1 |
SetClassWeight #
Weights C per class: a row's bound is C times its class's weight, so a weight above 1 makes errors on that class costlier.
method : public : SetClassWeight(negative:Float, positive:Float) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| negative | Float | weight for false labels; must be > 0 |
| positive | Float | weight for true labels; must be > 0 |
SetClassWeightBalanced #
Weights C per class by n / (2 * rows in the class) at Fit, scikit-learn's class_weight='balanced', so a rare class is not traded away.
method : public : SetClassWeightBalanced() ~ NilSetCoef0 #
Sets the independent term of the POLY and SIGMOID kernels.
method : public : SetCoef0(coef0:Float) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| coef0 | Float | offset |
SetDegree #
Sets the POLY degree.
method : public : SetDegree(degree:Int) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| degree | Int | degree; must be >= 0 |
SetGamma #
Sets an explicit gamma.
method : public : SetGamma(gamma:Float) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| gamma | Float | kernel coefficient; must be >= 0 |
SetGammaAuto #
Uses gamma = 1 / features at Fit, scikit-learn's 'auto'.
method : public : SetGammaAuto() ~ NilSetGammaScale #
Uses gamma = 1 / (features * X.var()) at Fit, scikit-learn's 'scale' and the default. X.var() is the population variance of every entry of X; when it is zero gamma is 1.0.
method : public : SetGammaScale() ~ NilSetMaxIterations #
Caps the solver's iterations. When the cap is reached Fit still returns true and the model is usable, but IsConverged is false.
method : public : SetMaxIterations(max_iter:Int) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| max_iter | Int | iteration cap; 0 or less means LIBSVM's own cap, max(10,000,000, 100 * rows) |