v2026.11.2
All Bundles
Bundle

Sentence embeddings from a pretrained BERT-family encoder exported to ONNX: the text is tokenized with WordPieceTokenizer, run through API.Onnx, and pooled into one vector per text.

A library of its own because it needs both System.NLP and API.Onnx, and the compiler resolves every class a linked library names: an embedder inside onnx.obl would make every ONNX program link nlp too, and one inside nlp.obl would make every NLP program need ONNX Runtime (docs/ML_BERT_DESIGN.md). Compile with -lib models,opencv,cipher,json,onnx,nlp,nlp_onnx.

BertEmbedder

Sentence embeddings from a BERT-family ONNX model: Embed(texts) gives one row of hidden-size numbers per text.

The model is a Hugging Face export (optimum-cli export onnx, or torch.onnx.export) taking input_ids, and usually attention_mask and token_type_ids, as INT64[batch, sequence], and giving last_hidden_state FP32[batch, sequence, hidden] and, for BERT though not for DistilBERT, pooler_output FP32[batch, hidden]. The model's own signature decides what is sent: an input it does not declare is not built, and the one output the pooling needs is the only one asked for.

With no session options, the build's default provider runs the model, and the CPU when that provider cannot open it (DirectML needs a display adapter): see IsCpuFallback.

Pooling: CLS takes the hidden state of the first token; MEAN averages the hidden states of the real tokens, padding excluded, as sentence-transformers does; POOLER is the model's pooler_output, when it has one.

Every text is padded to max_len, not to the longest text in its batch, so a text's embedding does not depend on which texts share its batch.

The encoder is frozen: this computes embeddings, it does not train the model. A classifier trained on them (System.ML's EmbeddingClassifier) is the frozen-encoder alternative to fine-tuning.

Example

embedder := BertEmbedder->New("model.onnx", "vocab.txt", 128);
if(<>embedder->IsOpen()) {
  embedder->GetError()->PrintLine();
  return;
};
embedder->SetPooling(BertEmbedder->Pooling->MEAN);
X := embedder->Embed(["first text", "second text"]);   # Float[2, hidden]
embedder->Close();

Operations

Close #

Releases the model. Safe to call more than once.

method : public : Close() ~ Nil

ClsPool # function

CLS pooling: for each row, the vector of the first token.

function : ClsPool(hidden:Float[], batch:Int, sequence:Int, dim:Int) ~ Float[,]

Parameters

NameTypeDescription
hiddenFloattoken vectors, flat and row-major: [batch, sequence, dim]
batchIntrows
sequenceInttokens per row
dimIntlength of each token vector

Return

TypeDescription
Float[batch, dim]

Embed #

Embeds texts with the configured pooling.

method : public : Embed(texts:String[]) ~ Float[,]

Parameters

NameTypeDescription
textsStringthe texts

Return

TypeDescription
Floatone row per text, hidden-size columns; Nil on failure (see GetError)

Embed #

Embeds texts with a given pooling.

method : public : Embed(texts:String[], pooling:BertEmbedder->Pooling) ~ Float[,]

Parameters

NameTypeDescription
textsStringthe texts
poolingBertEmbedder->PoolingCLS, MEAN or POOLER

Return

TypeDescription
Floatone row per text, hidden-size columns; Nil on failure (see GetError)

Embed #

Embeds texts with a given pooling and layer strategy.

method : public : Embed(texts:String[], pooling:BertEmbedder->Pooling, layers:BertEmbedder->Layers) ~ Float[,]

Parameters

NameTypeDescription
textsStringthe texts
poolingBertEmbedder->PoolingCLS or MEAN (POOLER only with LAST)
layersBertEmbedder->LayersLAST, SUM_LAST_4 or CONCAT_LAST_4

Return

TypeDescription
Floatone row per text, GetEmbeddingSize columns; Nil on failure (see GetError), e.g. when the model has no hidden layers

Example

X := embedder->Embed(texts, BertEmbedder->Pooling->MEAN, BertEmbedder->Layers->CONCAT_LAST_4);
if(X = Nil) {
  embedder->GetError()->PrintLine();
};

EmbedEncoded #

Embeds texts already encoded, with the configured pooling: for a caller with its own tokenizer or cached encodings. Every encoding must be max_len long.

method : public : EmbedEncoded(encodings:TokenEncoding[]) ~ Float[,]

Parameters

NameTypeDescription
encodingsTokenEncodingthe encoded texts

Return

TypeDescription
Floatone row per text, hidden-size columns; Nil on failure (see GetError)

EmbedEncoded #

Embeds texts already encoded, with a given pooling.

method : public : EmbedEncoded(encodings:TokenEncoding[], pooling:BertEmbedder->Pooling) ~ Float[,]

Parameters

NameTypeDescription
encodingsTokenEncodingthe encoded texts, each max_len long
poolingBertEmbedder->PoolingCLS, MEAN or POOLER

Return

TypeDescription
Floatone row per text, hidden-size columns; Nil on failure (see GetError)

EmbedEncoded #

Embeds texts already encoded, with a given pooling and layer strategy.

method : public : EmbedEncoded(encodings:TokenEncoding[], pooling:BertEmbedder->Pooling, layers:BertEmbedder->Layers) ~ Float[,]

Parameters

NameTypeDescription
encodingsTokenEncodingthe encoded texts, each max_len long
poolingBertEmbedder->PoolingCLS or MEAN (POOLER only with LAST)
layersBertEmbedder->LayersLAST, SUM_LAST_4 or CONCAT_LAST_4

Return

TypeDescription
Floatone row per text, GetEmbeddingSize columns; Nil on failure (see GetError)

GetBatchSize #

Texts run through the model at once.

method : public : GetBatchSize() ~ Int

Return

TypeDescription
Intbatch size

GetEmbeddingSize #

The length of each embedding for a layer strategy: the hidden size, or four times it for CONCAT_LAST_4.

method : public : GetEmbeddingSize(layers:BertEmbedder->Layers) ~ Int

Parameters

NameTypeDescription
layersBertEmbedder->Layersthe strategy

Return

TypeDescription
Intthe length, or -1 when the hidden size is not yet known

GetError #

Why the embedder did not open, or why the last Embed returned Nil.

method : public : GetError() ~ String

Return

TypeDescription
Stringthe reason, or Nil

GetHiddenSize #

The length of each embedding: the model's hidden size. A model whose hidden axis is symbolic reports it after the first Embed.

method : public : GetHiddenSize() ~ Int

Return

TypeDescription
Inthidden size, or -1 when not yet known

GetLayers #

The layer strategy Embed uses.

method : public : GetLayers() ~ BertEmbedder->Layers

Return

TypeDescription
BertEmbedder->Layerslayers

GetMaxLength #

The sequence length every text is truncated and padded to.

method : public : GetMaxLength() ~ Int

Return

TypeDescription
Intmax_len

GetPooling #

The pooling Embed uses.

method : public : GetPooling() ~ BertEmbedder->Pooling

Return

TypeDescription
BertEmbedder->Poolingpooling

GetSession #

The ONNX session, for what this class does not cover.

method : public : GetSession() ~ Session

Return

TypeDescription
Sessionthe session

GetTokenizer #

The tokenizer, e.g. to inspect what a text becomes.

method : public : GetTokenizer() ~ WordPieceTokenizer

Return

TypeDescription
WordPieceTokenizerthe tokenizer

HasHiddenLayers #

Whether the model exports its intermediate layers, so SUM_LAST_4 and CONCAT_LAST_4 are available: four or more FP32 outputs named hidden_states.N (or hidden_states_N, hidden_state_N), or one FP32 [layer, batch, sequence, hidden] output named hidden_states. A standard export does not have them; export with output_hidden_states=True and name the outputs so.

method : public : HasHiddenLayers() ~ Bool

Return

TypeDescription
Booltrue when the model has them

HasPooler #

Whether the model has a pooler_output, so POOLER pooling is available.

method : public : HasPooler() ~ Bool

Return

TypeDescription
Booltrue when it has one

IsCpuFallback #

Whether the build's default provider failed to open the model and the CPU provider was used instead. Only when no session options were given.

method : public : IsCpuFallback() ~ Bool

Return

TypeDescription
Booltrue when running on the CPU as a fallback

IsOpen #

Whether the model and vocabulary loaded and the model has the inputs and outputs of a BERT-family encoder.

method : public : IsOpen() ~ Bool

Return

TypeDescription
Booltrue when Embed can run

Load # function

Opens a model with the vocabulary of an uncased model.

function : Load(model_path:String, vocab_path:String, max_len:Int) ~ BertEmbedder

Parameters

NameTypeDescription
model_pathStringthe .onnx file
vocab_pathStringthe model's vocab.txt
max_lenIntthe sequence length every text is truncated and padded to

Return

TypeDescription
BertEmbedderthe embedder; test IsOpen before using it

MeanPool # function

Mean pooling: for each row, the mean of the token vectors whose mask is 1. A row with no real token pools to zeros.

function : MeanPool(hidden:Float[], batch:Int, sequence:Int, dim:Int, mask:Int[]) ~ Float[,]

Parameters

NameTypeDescription
hiddenFloattoken vectors, flat and row-major: [batch, sequence, dim]
batchIntrows
sequenceInttokens per row
dimIntlength of each token vector
maskIntattention mask, flat: [batch, sequence]

Return

TypeDescription
Float[batch, dim]

Example

pooled := BertEmbedder->MeanPool(values, 2, 128, 768, mask);

New # constructor

Opens a model with the vocabulary of an uncased model: lowercase, strip accents.

New(model_path:String, vocab_path:String, max_len:Int)

Parameters

NameTypeDescription
model_pathStringthe .onnx file
vocab_pathStringthe model's vocab.txt
max_lenIntthe sequence length every text is truncated and padded to

New # constructor

Opens a model with a tokenizer configured by the caller, and session options.

New(model_path:String, tokenizer:WordPieceTokenizer, max_len:Int, config:Map<String,String>)

Parameters

NameTypeDescription
model_pathStringthe .onnx file
tokenizerWordPieceTokenizerthe model's tokenizer, e.g. a cased one
max_lenIntthe sequence length every text is truncated and padded to
configMap<String,String>session options, e.g. "ep" to "cpu"; may be Nil

PoolLayers # function

Pools stacked hidden layers and combines them: for each of the layers, CLS or MEAN pooling as in ClsPool and MeanPool, then, for SUM_LAST_4, the element-wise sum of the layers' vectors (dim columns) or, for CONCAT_LAST_4, the layers' vectors one after another (layers x dim columns, the first layer first). LAST uses the final layer alone.

function : PoolLayers(stacked:Float[], layer_count:Int, batch:Int, sequence:Int, dim:Int, mask:Int[], pooling:BertEmbedder->Pooling, layers:BertEmbedder->Layers) ~ Float[,]

Parameters

NameTypeDescription
stackedFloatthe layers, flat: [layer, batch, sequence, dim], oldest layer first
layer_countIntlayers held in stacked, at least 1
batchIntrows
sequenceInttokens per row
dimIntlength of each token vector
maskIntattention mask, flat: [batch, sequence]
poolingBertEmbedder->PoolingCLS or MEAN
layersBertEmbedder->Layersthe strategy

Return

TypeDescription
Float[batch, dim] (LAST, SUM_LAST_4) or [batch, 4 x dim] (CONCAT_LAST_4); Nil when the arguments do not fit

Example

X := BertEmbedder->PoolLayers(stacked, 4, 2, 128, 768, mask, BertEmbedder->Pooling->MEAN, BertEmbedder->Layers->SUM_LAST_4);

SetBatchSize #

Sets how many texts run through the model at once. Larger batches are faster per text and hold more memory: a batch's hidden states are batch x max_len x hidden numbers. A model with a fixed batch axis caps it.

method : public : SetBatchSize(batch_size:Int) ~ Nil

Parameters

NameTypeDescription
batch_sizeInttexts per run, at least 1

SetLayers #

Sets the layer strategy Embed uses; LAST by default. SUM_LAST_4 and CONCAT_LAST_4 need HasHiddenLayers; without it Embed returns Nil with a reason.

method : public : SetLayers(layers:BertEmbedder->Layers) ~ Nil

Parameters

NameTypeDescription
layersBertEmbedder->LayersLAST, SUM_LAST_4 or CONCAT_LAST_4

SetPooling #

Sets the pooling Embed uses; CLS by default.

method : public : SetPooling(pooling:BertEmbedder->Pooling) ~ Nil

Parameters

NameTypeDescription
poolingBertEmbedder->PoolingCLS, MEAN or POOLER