Convolutional neural networks on the CPU: layers with forward and backward passes, the Adam and SGD optimizers, an image dataset with train, validation and test splits and on-the-fly augmentation, and a training loop that records the metrics of every epoch. The layers are native (libobjk_ml, Eigen plus a small thread pool); this library holds the model API, the dataset and the trainer. Compile with -lib ml_cnn,ml,ml_lab,opencv,csv,json,cipher,gen_collect. Design, validation and measurements: docs/AI.md, "Convolutional networks (System.ML.Cnn)".
CnnModel
A convolutional network for binary classification: a stack of layers, in NCHW float32, ending in one logit per image, trained with binary cross-entropy on logits and the Adam or SGD optimizer. Build it with the Add methods, then Compile.
Layers: Conv (stride, VALID or SAME padding, bias; im2col and a matrix multiply), MaxPool, ReLU, Flatten, Dense and Dropout (inverted: the survivors are scaled by 1 / (1 - rate) in training, and evaluation is the identity). The last layer must give one value per image, so a model ends in AddDense(1). Predict returns sigmoid of that logit, the probability of class 1.
Images are Float[] of whole images laid end to end, each channels x height x width (planes, not interleaved) with values in 0 .. 1; labels are 0.0 or 1.0. Nothing here ends the program on bad input: a method answers false, -1.0 or Nil (or 0 for a count), and GetError() says why.
A model holds native memory, so call Close() when done with it (it is safe to call twice, and IsOpen() says whether the model is still alive). Not thread-safe: one thread at a time per model; the layers use the process-wide pool of CnnRuntime, whose thread count also decides whether a run repeats bit for bit.
Example
model := CnnModel->New(3, 64, 64, 42);
model->AddConv(16, 3, 1, Padding->VALID);
model->AddReLU();
model->AddMaxPool(2);
model->AddFlatten();
model->AddDense(1);
model->Compile(0.001);
loss := model->TrainBatch(images, labels);
probabilities := model->Predict(images);Operations
- New
- NewReference
- AddConv
- AddDense
- AddDropout
- AddFlatten
- AddMaxPool
- AddReLU
- Baseline
- Close
- Compile
- CompileAdam
- CompileSgd
- EvaluateBatch
- EvaluateStaged
- GetActivation
- GetChannels
- GetCorrect
- GetDevice
- GetDeviceMemory
- GetError
- GetGradients
- GetHeight
- GetInputGradient
- GetParameterCount
- GetSummary
- GetWeights
- GetWidth
- Gradients
- GradientsTraining
- GradientsWithInput
- IsCompiled
- IsOpen
- IsValid
- Load
- Predict
- PredictLogits
- PredictStaged
- ResetDropout
- Save
- SetDevice
- SetInitializer
- SetLearningRate
- SetTraining
- SetWeights
- StageImage
- TrainBatch
- TrainStaged
AddConv #
Adds a 2D convolution (cross-correlation) with a square kernel and a bias per filter.
method : public : AddConv(filters:Int, kernel:Int, stride:Int, padding:Padding) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| filters | Int | output channels |
| kernel | Int | kernel height and width |
| stride | Int | step between windows |
| padding | Padding | VALID or SAME |
Return
| Type | Description |
|---|---|
| Bool | true if added, false if refused (see GetError) |
AddConv #
Adds a convolution with stride 1 and no padding.
method : public : AddConv(filters:Int, kernel:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| filters | Int | output channels |
| kernel | Int | kernel height and width |
Return
| Type | Description |
|---|---|
| Bool | true if added |
AddDense #
Adds a fully connected layer. Any input shape is read as one flat vector per image.
method : public : AddDense(units:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| units | Int | output width |
Return
| Type | Description |
|---|---|
| Bool | true if added |
AddDropout #
Adds inverted dropout. Its masks come from the model seed, so training repeats exactly.
method : public : AddDropout(rate:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| rate | Float | fraction of activations to zero, in [0, 1) |
Return
| Type | Description |
|---|---|
| Bool | true if added |
AddFlatten #
Adds a flatten layer: channels x height x width become one vector per image.
method : public : AddFlatten() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true if added |
AddMaxPool #
Adds max pooling over size x size windows that step by the window size.
method : public : AddMaxPool(size:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| size | Int | window size |
Return
| Type | Description |
|---|---|
| Bool | true if added |
AddMaxPool #
Adds max pooling.
method : public : AddMaxPool(size:Int, stride:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| size | Int | window size |
| stride | Int | step between windows |
Return
| Type | Description |
|---|---|
| Bool | true if added |
AddReLU #
Adds a ReLU activation.
method : public : AddReLU() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true if added |
Baseline # function
The baseline of the deepfake-detection study, compiled and ready to train: conv_blocks blocks of Conv(3x3, VALID) + ReLU + MaxPool(2) with 16, 32, 64, ... filters, then Flatten, Dense(dense_units) + ReLU, Dropout and Dense(1), trained with Adam. At 224 x 224 with three blocks and 128 dense units it has 5,561,633 parameters.
function : Baseline(image_size:Int, conv_blocks:Int, dropout:Float, dense_units:Int, learning_rate:Float, seed:Int) ~ CnnModelParameters
| Name | Type | Description |
|---|---|---|
| image_size | Int | images are image_size x image_size RGB |
| conv_blocks | Int | number of convolution blocks, 1 or more |
| dropout | Float | dropout rate in [0, 1) |
| dense_units | Int | width of the hidden dense layer |
| learning_rate | Float | Adam learning rate |
| seed | Int | seed |
Return
| Type | Description |
|---|---|
| CnnModel | the model, or Nil when the architecture does not fit the image size |
Close #
Releases the native model. The object is not usable afterwards; every method answers as for an invalid model.
method : public : Close() ~ NilCompile #
Initializes the weights from the seed and sets the Adam optimizer (beta1 0.9, beta2 0.999, epsilon 1e-8, as in PyTorch and Keras).
method : public : Compile(learning_rate:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| learning_rate | Float | step size, above 0 |
Return
| Type | Description |
|---|---|
| Bool | true if compiled, false if the model is empty, does not end in one value per image, or was compiled already |
CompileAdam #
Compiles with Adam and the given moment decays.
method : public : CompileAdam(learning_rate:Float, beta1:Float, beta2:Float, epsilon:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| learning_rate | Float | step size, above 0 |
| beta1 | Float | first-moment decay, [0, 1) |
| beta2 | Float | second-moment decay, [0, 1) |
| epsilon | Float | added to the denominator, above 0 |
Return
| Type | Description |
|---|---|
| Bool | true if compiled |
CompileSgd #
Compiles with stochastic gradient descent.
method : public : CompileSgd(learning_rate:Float, momentum:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| learning_rate | Float | step size, above 0 |
| momentum | Float | momentum in [0, 1), 0.0 for plain SGD |
Return
| Type | Description |
|---|---|
| Bool | true if compiled |
EvaluateBatch #
The mean loss of a batch in evaluation mode, without changing the model.
method : public : EvaluateBatch(images:Float[], labels:Float[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| images | Float | the batch |
| labels | Float | 0.0 or 1.0 per image |
Return
| Type | Description |
|---|---|
| Float | loss, or -1.0 if refused |
EvaluateStaged #
EvaluateBatch on the staged images.
method : public : EvaluateStaged(labels:Float[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| labels | Float | one per staged image |
Return
| Type | Description |
|---|---|
| Float | the loss, or -1.0 if refused |
GetActivation #
The output of a layer for the last forward pass over the whole batch.
method : public : GetActivation(layer:Int) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| layer | Int | layer index, from 0 |
Return
| Type | Description |
|---|---|
| Float | the activations, or Nil if there is no such layer or no pass yet |
GetChannels #
Gets the input channels.
method : public : GetChannels() ~ IntReturn
| Type | Description |
|---|---|
| Int | channels |
GetCorrect #
How many images of the last TrainBatch, EvaluateBatch, TrainStaged or EvaluateStaged call were classified correctly at the 0.5 threshold.
method : public : GetCorrect() ~ IntReturn
| Type | Description |
|---|---|
| Int | count |
GetDevice #
Where the model runs.
method : public : GetDevice() ~ StringReturn
| Type | Description |
|---|---|
| String | "cpu" or "cuda:N", "cpu" for a model that is not valid |
GetDeviceMemory #
The GPU memory this model holds (parameters, optimizer state, and the activation buffers of the largest batch it has seen). Grows with the batch size; all of it is given back by Close().
method : public : GetDeviceMemory() ~ IntReturn
| Type | Description |
|---|---|
| Int | bytes, 0 on the CPU |
GetError #
Why the last call answered false, -1.0 or Nil.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, empty before any refusal |
GetGradients #
The gradient of the last Gradients, GradientsTraining, GradientsWithInput or training call, in the layout of GetWeights.
method : public : GetGradients() ~ Float[]Return
| Type | Description |
|---|---|
| Float | the gradients, or Nil before any pass |
GetInputGradient #
The gradient of the loss with respect to the input pixels, after GradientsWithInput.
method : public : GetInputGradient() ~ Float[]Return
| Type | Description |
|---|---|
| Float | the gradient, one value per input value, or Nil |
GetParameterCount #
The number of trainable parameters: for a convolution filters x (channels x k x k) + filters, for a dense layer units x inputs + units.
method : public : GetParameterCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | parameter count, 0 for a model that is not valid |
GetSummary #
A listing of the layers with output shapes and parameter counts.
method : public : GetSummary() ~ StringReturn
| Type | Description |
|---|---|
| String | the summary |
GetWeights #
Every trainable value in one array, layer by layer, each layer's weights then its biases. A convolution's weights are filters x channels x kh x kw and a dense layer's are units x inputs, both row-major (the layouts of PyTorch).
method : public : GetWeights() ~ Float[]Return
| Type | Description |
|---|---|
| Float | the values, or Nil if the model is not compiled |
Gradients #
Forward and backward pass without the update, in evaluation mode; GetGradients then returns the parameter gradients. Used to check backpropagation.
method : public : Gradients(images:Float[], labels:Float[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| images | Float | the batch |
| labels | Float | 0.0 or 1.0 per image |
Return
| Type | Description |
|---|---|
| Float | the loss, or -1.0 if refused |
GradientsTraining #
Forward and backward pass without the update, in training mode (dropout on, drawing the next masks); see ResetDropout.
method : public : GradientsTraining(images:Float[], labels:Float[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| images | Float | the batch |
| labels | Float | 0.0 or 1.0 per image |
Return
| Type | Description |
|---|---|
| Float | the loss, or -1.0 if refused |
GradientsWithInput #
Like Gradients, in evaluation mode, and also keeps the gradient of the loss with respect to the input pixels for GetInputGradient.
method : public : GradientsWithInput(images:Float[], labels:Float[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| images | Float | the batch |
| labels | Float | 0.0 or 1.0 per image |
Return
| Type | Description |
|---|---|
| Float | the loss, or -1.0 if refused |
IsCompiled #
Is the model compiled?
method : public : IsCompiled() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true if so |
IsOpen #
Is the native model alive: created and not yet closed? The word BertEmbedder and Session use; the same as IsValid.
method : public : IsOpen() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true if open |
IsValid #
Is the model usable (was the native model created)?
method : public : IsValid() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true if valid |
Load #
Reads weights back into this model, which must be compiled and have the same architecture and scalar type as the one that wrote the file. A file from another architecture, a truncated file or a file that is not a weights file is refused and leaves the model unchanged.
method : public : Load(path:String) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| path | String | file to read |
Return
| Type | Description |
|---|---|
| Bool | true if loaded |
New # constructor
Creates an empty float32 model.
New(channels:Int, height:Int, width:Int, seed:Int)Parameters
| Name | Type | Description |
|---|---|---|
| channels | Int | input channels: 1 (grayscale) or 3 (RGB) for images staged from files or pixels, up to 4096 for Float[] batches |
| height | Int | image height |
| width | Int | image width |
| seed | Int | seed of the weights and of the dropout masks |
NewReference # function
Creates an empty float64 model. It is the same code in double precision, slower, meant for checking gradients: in float32 a central difference cannot resolve a gradient better than about 1e-3, in float64 it agrees with backpropagation to about 1e-8.
function : NewReference(channels:Int, height:Int, width:Int, seed:Int) ~ CnnModelParameters
| Name | Type | Description |
|---|---|---|
| channels | Int | input channels |
| height | Int | image height |
| width | Int | image width |
| seed | Int | seed of the weights and of the dropout masks |
Return
| Type | Description |
|---|---|
| CnnModel | the model |
Predict #
The probability of class 1 for each image of a batch (sigmoid of the logit). Dropout is off unless SetTraining(true) was called.
method : public : Predict(images:Float[]) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| images | Float | whole images end to end, any number of them |
Return
| Type | Description |
|---|---|
| Float | one probability per image, or Nil if refused (not compiled, or not a whole number of images) |
PredictLogits #
The raw logit for each image of a batch.
method : public : PredictLogits(images:Float[]) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| images | Float | whole images end to end |
Return
| Type | Description |
|---|---|
| Float | one logit per image, or Nil if refused |
PredictStaged #
Predicts the staged images.
method : public : PredictStaged(count:Int, logits:Bool) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| count | Int | how many images were staged |
| logits | Bool | true for logits, false for probabilities |
Return
| Type | Description |
|---|---|
| Float | one value per image, or Nil if refused |
ResetDropout #
Restarts the dropout masks, so the next forward pass draws the same masks as the first one did. Gradient checks need this to evaluate one loss twice with one mask.
method : public : ResetDropout() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true if done |
Save #
Writes the weights to a file: a header with the architecture's signature, then every value exactly as it is held, so Load gives back bit-identical weights. The optimizer's moments are not saved.
method : public : Save(path:String) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| path | String | file to write |
Return
| Type | Description |
|---|---|
| Bool | true if written |
SetDevice #
Chooses where the model runs: "cpu" (the default) or "cuda" / "cuda:N", the GPU backend. Call it before training, or at any time: the weights move with the model (the Adam moments start again from zero). The two devices share the weights file format, so a model saved on one loads on the other. A refusal (no GPU, no CUDA library, out of GPU memory) answers false, says why in GetError() and leaves the model where it was.
method : public : SetDevice(device:String) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| device | String | "cpu", "cuda" or "cuda:N" |
Return
| Type | Description |
|---|---|
| Bool | true if the model now runs on that device |
SetDevice #
Chooses where the model runs, optionally with TF32 (the GPU's Tensor Cores, about 10 bits of mantissa in the matrix multiplies: faster, results a few 1e-4 from the exact FP32 of the default). Ignored on the CPU.
method : public : SetDevice(device:String, tf32:Bool) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| device | String | "cpu", "cuda" or "cuda:N" |
| tf32 | Bool | true to allow TF32 on the GPU |
Return
| Type | Description |
|---|---|
| Bool | true if the model now runs on that device |
SetInitializer #
Chooses how the weights start. Call it before Compile; it has no effect afterwards.
method : public : SetInitializer(init:Initializer) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| init | Initializer | GLOROT (the default) or KAIMING |
SetLearningRate #
Changes the learning rate of the optimizer, keeping its moments.
method : public : SetLearningRate(learning_rate:Float) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| learning_rate | Float | step size, above 0 |
Return
| Type | Description |
|---|---|
| Bool | true if changed |
SetTraining #
Switches the mode Predict and the staged predictions run in. In training mode dropout is active; in evaluation mode (the default) it is the identity. TrainBatch and TrainStaged always train, and EvaluateBatch always evaluates.
method : public : SetTraining(training:Bool) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| training | Bool | true for training mode |
Return
| Type | Description |
|---|---|
| Bool | true if set |
SetWeights #
Replaces every trainable value; the array must have the layout and length GetWeights returns.
method : public : SetWeights(weights:Float[]) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| weights | Float | the values |
Return
| Type | Description |
|---|---|
| Bool | true if set, false if the length is wrong |
StageImage #
Decodes one 8-bit image into slot `slot` of the staging batch: BGR (as OpenCV holds it) to RGB, scaled to 0 .. 1, and augmented if the configuration says so. Staging avoids building a Float[] of the whole batch in Objeck. Stage `total` images into slots 0 .. total - 1, then call TrainStaged, EvaluateStaged or PredictStaged.
method : public : StageImage(pixels:Byte[], rows:Int, cols:Int, slot:Int, total:Int, aug:Augmentation, key:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| pixels | Byte | rows x cols x channels bytes, interleaved, channels 1, 3 or 4 |
| rows | Int | image rows, must equal the model height |
| cols | Int | image columns, must equal the model width |
| slot | Int | index in the batch |
| total | Int | images in the batch |
| aug | Augmentation | the augmentation, or Nil for none |
| key | Int | distinguishes this image's random draws, see Augmentation |
Return
| Type | Description |
|---|---|
| Bool | true if staged |
TrainBatch #
One optimizer step on a batch: forward pass in training mode (dropout on), loss, backward pass, update.
method : public : TrainBatch(images:Float[], labels:Float[]) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| images | Float | the batch, whole images of channels x height x width values end to end |
| labels | Float | 0.0 or 1.0 per image |
Return
| Type | Description |
|---|---|
| Float | the mean loss of the batch before the update, or -1.0 if refused (an empty batch, a wrong size, not compiled) |