Sentence embeddings from a pretrained BERT-family encoder exported to ONNX: the text is tokenized with WordPieceTokenizer, run through API.Onnx, and pooled into one vector per text.
A library of its own because it needs both System.NLP and API.Onnx, and the compiler resolves every class a linked library names: an embedder inside onnx.obl would make every ONNX program link nlp too, and one inside nlp.obl would make every NLP program need ONNX Runtime (docs/ML_BERT_DESIGN.md). Compile with -lib models,opencv,cipher,json,onnx,nlp,nlp_onnx.
TextClassifier
Text classification with a fine-tuned BERT-family model, as Python's pipeline("text-classification"): a sentiment model, a topic classifier, a malware classifier fine-tuned on Colab.
The model is a Hugging Face model folder (what save_pretrained writes, with an ONNX export beside it), or a Hub repository's files downloaded as they are:
- the model: model.onnx, else onnx/model.onnx; or the file the "model" option names, relative to the folder; - the tokenizer: tokenizer.json (WordPiece), else vocab.txt with tokenizer_config.json's do_lower_case (true when absent, as BertTokenizer defaults); - the labels: config.json's id2label, else LABEL_0, LABEL_1, ...
The model's own signature decides what is sent: input_ids always, attention_mask and token_type_ids only when declared (DistilBERT and RoBERTa exports have no token_type_ids). The output is logits FP32[batch, labels].
Scores follow transformers: SIGMOID when config.json's problem_type is multi_label_classification or there is one label, else SOFTMAX. SetFunction overrides it.
Texts longer than the maximum length are truncated, and the Prediction says so. The maximum length is the smallest of 512, the model's max_position_embeddings and the tokenizer's model_max_length, or the model's fixed sequence axis. Each batch is padded to its longest text. With an fp32 model a text's scores do not depend on which texts share its batch. With a dynamically quantized one (model_quantized.onnx, model_int8.onnx) they do, slightly: activations are quantized with a scale taken from the whole batch. SetBatchSize(1) gives what Python's pipeline gives one text at a time.
With no session options, the build's default provider runs the model, and the CPU when that provider cannot open it: see IsCpuFallback. Any option other than "model" is a session option, e.g. "ep" to "cpu".
Example
classifier := TextClassifier->New("distilbert-sst2");
if(<>classifier->IsOpen()) {
classifier->GetError()->ErrorLine();
return;
};
each(p in classifier->Classify(["I loved it", "Not worth the ticket"])) {
p->ToString()->PrintLine(); # POSITIVE 0.9998, then NEGATIVE 0.9991
};
classifier->Close();Operations
- New
- Classify
- Close
- GetBatchSize
- GetError
- GetFunction
- GetLabels
- GetMaxLength
- GetModelPath
- GetSession
- GetTokenizer
- IsCpuFallback
- IsOpen
- Logits
- Scores
- SetBatchSize
- SetFunction
- SetMaxLength
Classify #
Classifies one text.
method : public : Classify(text:String) ~ PredictionParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Prediction | the prediction; Nil on failure (see GetError) |
Classify #
Classifies texts.
method : public : Classify(texts:String[]) ~ Prediction[]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
Return
| Type | Description |
|---|---|
| Prediction | one prediction per text; Nil on failure (see GetError) |
GetBatchSize #
Texts run through the model at once.
method : public : GetBatchSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | batch size |
GetError #
Why the classifier did not open, or why the last call returned Nil.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or Nil |
GetFunction #
How logits become scores.
method : public : GetFunction() ~ TextClassifier->FunctionReturn
| Type | Description |
|---|---|
| TextClassifier->Function | SOFTMAX, SIGMOID or NONE |
GetLabels #
The labels, in the order of the model's logits.
method : public : GetLabels() ~ String[]Return
| Type | Description |
|---|---|
| String | labels |
GetMaxLength #
The length, in tokens, a text is truncated to, [CLS] and [SEP] included.
method : public : GetMaxLength() ~ IntReturn
| Type | Description |
|---|---|
| Int | maximum length |
GetModelPath #
The ONNX file that was opened.
method : public : GetModelPath() ~ StringReturn
| Type | Description |
|---|---|
| String | path |
GetSession #
The ONNX session, for what this class does not cover.
method : public : GetSession() ~ SessionReturn
| Type | Description |
|---|---|
| Session | the session |
GetTokenizer #
The tokenizer, e.g. to inspect what a text becomes.
method : public : GetTokenizer() ~ WordPieceTokenizerReturn
| Type | Description |
|---|---|
| WordPieceTokenizer | the tokenizer |
IsCpuFallback #
Whether the build's default provider failed to open or to run the model and the CPU provider was used instead. Only when no session options were given.
method : public : IsCpuFallback() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when running on the CPU as a fallback |
IsOpen #
Whether the model, tokenizer and labels loaded.
method : public : IsOpen() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when Classify can run |
Logits #
The raw logits for each text.
method : public : Logits(texts:String[]) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
Return
| Type | Description |
|---|---|
| Float | [texts, labels]; Nil on failure (see GetError) |
New # constructor
Opens a model folder.
New(folder:String)Parameters
| Name | Type | Description |
|---|---|---|
| folder | String | the folder holding the model, tokenizer and config.json |
New # constructor
Opens a model folder with options: "model" for the ONNX file inside the folder (e.g. onnx/model_quantized.onnx); anything else is a session option, e.g. "ep" to "cpu".
New(folder:String, options:Map<String,String>)Parameters
| Name | Type | Description |
|---|---|---|
| folder | String | the folder holding the model, tokenizer and config.json |
| options | Map<String,String> | options; may be Nil |
Scores #
Every label's score for each text, with the configured function.
method : public : Scores(texts:String[]) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
Return
| Type | Description |
|---|---|
| Float | [texts, labels]; Nil on failure (see GetError) |
SetBatchSize #
Sets how many texts run through the model at once. A model with a fixed batch axis caps it. With a dynamically quantized model, 1 reproduces one-at-a-time results.
method : public : SetBatchSize(batch_size:Int) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| batch_size | Int | texts per run, at least 1 |
SetFunction #
Sets how logits become scores, overriding what config.json implies.
method : public : SetFunction(scoring:TextClassifier->Function) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| scoring | TextClassifier->Function | SOFTMAX, SIGMOID or NONE |
SetMaxLength #
Sets the length, in tokens, a text is truncated to, e.g. the MAX_SEQ_LENGTH a model was fine-tuned with. A model with a fixed sequence axis keeps its own.
method : public : SetMaxLength(max_len:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| max_len | Int | tokens, at least 2 |
Return
| Type | Description |
|---|---|
| Bool | true when set |