v2026.11.2
All Bundles
Bundle

Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.

SubwordTokenizer

A tokenizer that turns text into a model's input: WordPieceTokenizer (BERT, DistilBERT, MiniLM) and BpeTokenizer (RoBERTa, GPT-2). The task classes of System.NLP.Onnx take either.

Operations

Encode # virtual

Encodes one text, truncated and padded to max_len, special tokens included.

method : public : virtual : Encode(text:String, max_len:Int) ~ TokenEncoding

Parameters

NameTypeDescription
textStringthe text
max_lenIntthe length to truncate and pad to

Return

TypeDescription
TokenEncodingthe encoding, or Nil (see GetError)

EncodePair # virtual

Encodes a pair of texts, truncated and padded to max_len, special tokens included.

method : public : virtual : EncodePair(text_a:String, text_b:String, max_len:Int) ~ TokenEncoding

Parameters

NameTypeDescription
text_aStringthe first text
text_bStringthe second text
max_lenIntthe length to truncate and pad to

Return

TypeDescription
TokenEncodingthe encoding, or Nil (see GetError)

GetError # virtual

Why the vocabulary did not load, or why the last call returned Nil.

method : public : virtual : GetError() ~ String

Return

TypeDescription
Stringthe reason, or Nil

IsLoaded # virtual

Whether the vocabulary loaded.

method : public : virtual : IsLoaded() ~ Bool

Return

TypeDescription
Booltrue when the tokenizer can encode