Sentence embeddings from a pretrained BERT-family encoder exported to ONNX: the text is tokenized with WordPieceTokenizer, run through API.Onnx, and pooled into one vector per text.
A library of its own because it needs both System.NLP and API.Onnx, and the compiler resolves every class a linked library names: an embedder inside onnx.obl would make every ONNX program link nlp too, and one inside nlp.obl would make every NLP program need ONNX Runtime (docs/ML_BERT_DESIGN.md). Compile with -lib models,opencv,cipher,json,onnx,nlp,nlp_onnx.
BertEmbedder
Sentence embeddings from a BERT-family ONNX model: Embed(texts) gives one row of hidden-size numbers per text.
The model is a Hugging Face export (optimum-cli export onnx, or torch.onnx.export) taking input_ids, and usually attention_mask and token_type_ids, as INT64[batch, sequence], and giving last_hidden_state FP32[batch, sequence, hidden] and, for BERT though not for DistilBERT, pooler_output FP32[batch, hidden]. The model's own signature decides what is sent: an input it does not declare is not built, and the one output the pooling needs is the only one asked for.
With no session options, the build's default provider runs the model, and the CPU when that provider cannot open it (DirectML needs a display adapter): see IsCpuFallback.
Pooling: CLS takes the hidden state of the first token; MEAN averages the hidden states of the real tokens, padding excluded, as sentence-transformers does; POOLER is the model's pooler_output, when it has one.
Every text is padded to max_len, not to the longest text in its batch, so a text's embedding does not depend on which texts share its batch.
The encoder is frozen: this computes embeddings, it does not train the model. A classifier trained on them (System.ML's EmbeddingClassifier) is the frozen-encoder alternative to fine-tuning.
Example
embedder := BertEmbedder->New("model.onnx", "vocab.txt", 128);
if(<>embedder->IsOpen()) {
embedder->GetError()->PrintLine();
return;
};
embedder->SetPooling(BertEmbedder->Pooling->MEAN);
X := embedder->Embed(["first text", "second text"]); # Float[2, hidden]
embedder->Close();Operations
- New
- Close
- ClsPool
- Embed
- EmbedEncoded
- GetBatchSize
- GetEmbeddingSize
- GetError
- GetHiddenSize
- GetLayers
- GetMaxLength
- GetPooling
- GetSession
- GetTokenizer
- HasHiddenLayers
- HasPooler
- IsCpuFallback
- IsOpen
- Load
- MeanPool
- PoolLayers
- SetBatchSize
- SetLayers
- SetPooling
ClsPool # function
CLS pooling: for each row, the vector of the first token.
function : ClsPool(hidden:Float[], batch:Int, sequence:Int, dim:Int) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| hidden | Float | token vectors, flat and row-major: [batch, sequence, dim] |
| batch | Int | rows |
| sequence | Int | tokens per row |
| dim | Int | length of each token vector |
Return
| Type | Description |
|---|---|
| Float | [batch, dim] |
Embed #
Embeds texts with the configured pooling.
method : public : Embed(texts:String[]) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
Return
| Type | Description |
|---|---|
| Float | one row per text, hidden-size columns; Nil on failure (see GetError) |
Embed #
Embeds texts with a given pooling.
method : public : Embed(texts:String[], pooling:BertEmbedder->Pooling) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
| pooling | BertEmbedder->Pooling | CLS, MEAN or POOLER |
Return
| Type | Description |
|---|---|
| Float | one row per text, hidden-size columns; Nil on failure (see GetError) |
Embed #
Embeds texts with a given pooling and layer strategy.
method : public : Embed(texts:String[], pooling:BertEmbedder->Pooling, layers:BertEmbedder->Layers) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
| pooling | BertEmbedder->Pooling | CLS or MEAN (POOLER only with LAST) |
| layers | BertEmbedder->Layers | LAST, SUM_LAST_4 or CONCAT_LAST_4 |
Return
| Type | Description |
|---|---|
| Float | one row per text, GetEmbeddingSize columns; Nil on failure (see GetError), e.g. when the model has no hidden layers |
Example
X := embedder->Embed(texts, BertEmbedder->Pooling->MEAN, BertEmbedder->Layers->CONCAT_LAST_4);
if(X = Nil) {
embedder->GetError()->PrintLine();
};EmbedEncoded #
Embeds texts already encoded, with the configured pooling: for a caller with its own tokenizer or cached encodings. Every encoding must be max_len long.
method : public : EmbedEncoded(encodings:TokenEncoding[]) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| encodings | TokenEncoding | the encoded texts |
Return
| Type | Description |
|---|---|
| Float | one row per text, hidden-size columns; Nil on failure (see GetError) |
EmbedEncoded #
Embeds texts already encoded, with a given pooling.
method : public : EmbedEncoded(encodings:TokenEncoding[], pooling:BertEmbedder->Pooling) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| encodings | TokenEncoding | the encoded texts, each max_len long |
| pooling | BertEmbedder->Pooling | CLS, MEAN or POOLER |
Return
| Type | Description |
|---|---|
| Float | one row per text, hidden-size columns; Nil on failure (see GetError) |
EmbedEncoded #
Embeds texts already encoded, with a given pooling and layer strategy.
method : public : EmbedEncoded(encodings:TokenEncoding[], pooling:BertEmbedder->Pooling, layers:BertEmbedder->Layers) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| encodings | TokenEncoding | the encoded texts, each max_len long |
| pooling | BertEmbedder->Pooling | CLS or MEAN (POOLER only with LAST) |
| layers | BertEmbedder->Layers | LAST, SUM_LAST_4 or CONCAT_LAST_4 |
Return
| Type | Description |
|---|---|
| Float | one row per text, GetEmbeddingSize columns; Nil on failure (see GetError) |
GetBatchSize #
Texts run through the model at once.
method : public : GetBatchSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | batch size |
GetEmbeddingSize #
The length of each embedding for a layer strategy: the hidden size, or four times it for CONCAT_LAST_4.
method : public : GetEmbeddingSize(layers:BertEmbedder->Layers) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| layers | BertEmbedder->Layers | the strategy |
Return
| Type | Description |
|---|---|
| Int | the length, or -1 when the hidden size is not yet known |
GetError #
Why the embedder did not open, or why the last Embed returned Nil.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or Nil |
GetHiddenSize #
The length of each embedding: the model's hidden size. A model whose hidden axis is symbolic reports it after the first Embed.
method : public : GetHiddenSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | hidden size, or -1 when not yet known |
GetLayers #
The layer strategy Embed uses.
method : public : GetLayers() ~ BertEmbedder->LayersReturn
| Type | Description |
|---|---|
| BertEmbedder->Layers | layers |
GetMaxLength #
The sequence length every text is truncated and padded to.
method : public : GetMaxLength() ~ IntReturn
| Type | Description |
|---|---|
| Int | max_len |
GetPooling #
The pooling Embed uses.
method : public : GetPooling() ~ BertEmbedder->PoolingReturn
| Type | Description |
|---|---|
| BertEmbedder->Pooling | pooling |
GetSession #
The ONNX session, for what this class does not cover.
method : public : GetSession() ~ SessionReturn
| Type | Description |
|---|---|
| Session | the session |
GetTokenizer #
The tokenizer, e.g. to inspect what a text becomes.
method : public : GetTokenizer() ~ WordPieceTokenizerReturn
| Type | Description |
|---|---|
| WordPieceTokenizer | the tokenizer |
HasHiddenLayers #
Whether the model exports its intermediate layers, so SUM_LAST_4 and CONCAT_LAST_4 are available: four or more FP32 outputs named hidden_states.N (or hidden_states_N, hidden_state_N), or one FP32 [layer, batch, sequence, hidden] output named hidden_states. A standard export does not have them; export with output_hidden_states=True and name the outputs so.
method : public : HasHiddenLayers() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when the model has them |
HasPooler #
Whether the model has a pooler_output, so POOLER pooling is available.
method : public : HasPooler() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when it has one |
IsCpuFallback #
Whether the build's default provider failed to open the model and the CPU provider was used instead. Only when no session options were given.
method : public : IsCpuFallback() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when running on the CPU as a fallback |
IsOpen #
Whether the model and vocabulary loaded and the model has the inputs and outputs of a BERT-family encoder.
method : public : IsOpen() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when Embed can run |
Load # function
Opens a model with the vocabulary of an uncased model.
function : Load(model_path:String, vocab_path:String, max_len:Int) ~ BertEmbedderParameters
| Name | Type | Description |
|---|---|---|
| model_path | String | the .onnx file |
| vocab_path | String | the model's vocab.txt |
| max_len | Int | the sequence length every text is truncated and padded to |
Return
| Type | Description |
|---|---|
| BertEmbedder | the embedder; test IsOpen before using it |
MeanPool # function
Mean pooling: for each row, the mean of the token vectors whose mask is 1. A row with no real token pools to zeros.
function : MeanPool(hidden:Float[], batch:Int, sequence:Int, dim:Int, mask:Int[]) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| hidden | Float | token vectors, flat and row-major: [batch, sequence, dim] |
| batch | Int | rows |
| sequence | Int | tokens per row |
| dim | Int | length of each token vector |
| mask | Int | attention mask, flat: [batch, sequence] |
Return
| Type | Description |
|---|---|
| Float | [batch, dim] |
Example
pooled := BertEmbedder->MeanPool(values, 2, 128, 768, mask);New # constructor
Opens a model with the vocabulary of an uncased model: lowercase, strip accents.
New(model_path:String, vocab_path:String, max_len:Int)Parameters
| Name | Type | Description |
|---|---|---|
| model_path | String | the .onnx file |
| vocab_path | String | the model's vocab.txt |
| max_len | Int | the sequence length every text is truncated and padded to |
New # constructor
Opens a model with a tokenizer configured by the caller, and session options.
New(model_path:String, tokenizer:WordPieceTokenizer, max_len:Int, config:Map<String,String>)Parameters
| Name | Type | Description |
|---|---|---|
| model_path | String | the .onnx file |
| tokenizer | WordPieceTokenizer | the model's tokenizer, e.g. a cased one |
| max_len | Int | the sequence length every text is truncated and padded to |
| config | Map<String,String> | session options, e.g. "ep" to "cpu"; may be Nil |
PoolLayers # function
Pools stacked hidden layers and combines them: for each of the layers, CLS or MEAN pooling as in ClsPool and MeanPool, then, for SUM_LAST_4, the element-wise sum of the layers' vectors (dim columns) or, for CONCAT_LAST_4, the layers' vectors one after another (layers x dim columns, the first layer first). LAST uses the final layer alone.
function : PoolLayers(stacked:Float[], layer_count:Int, batch:Int, sequence:Int, dim:Int, mask:Int[], pooling:BertEmbedder->Pooling, layers:BertEmbedder->Layers) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| stacked | Float | the layers, flat: [layer, batch, sequence, dim], oldest layer first |
| layer_count | Int | layers held in stacked, at least 1 |
| batch | Int | rows |
| sequence | Int | tokens per row |
| dim | Int | length of each token vector |
| mask | Int | attention mask, flat: [batch, sequence] |
| pooling | BertEmbedder->Pooling | CLS or MEAN |
| layers | BertEmbedder->Layers | the strategy |
Return
| Type | Description |
|---|---|
| Float | [batch, dim] (LAST, SUM_LAST_4) or [batch, 4 x dim] (CONCAT_LAST_4); Nil when the arguments do not fit |
Example
X := BertEmbedder->PoolLayers(stacked, 4, 2, 128, 768, mask, BertEmbedder->Pooling->MEAN, BertEmbedder->Layers->SUM_LAST_4);SetBatchSize #
Sets how many texts run through the model at once. Larger batches are faster per text and hold more memory: a batch's hidden states are batch x max_len x hidden numbers. A model with a fixed batch axis caps it.
method : public : SetBatchSize(batch_size:Int) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| batch_size | Int | texts per run, at least 1 |
SetLayers #
Sets the layer strategy Embed uses; LAST by default. SUM_LAST_4 and CONCAT_LAST_4 need HasHiddenLayers; without it Embed returns Nil with a reason.
method : public : SetLayers(layers:BertEmbedder->Layers) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| layers | BertEmbedder->Layers | LAST, SUM_LAST_4 or CONCAT_LAST_4 |
SetPooling #
Sets the pooling Embed uses; CLS by default.
method : public : SetPooling(pooling:BertEmbedder->Pooling) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| pooling | BertEmbedder->Pooling | CLS, MEAN or POOLER |