v2026.11.2
All Bundles
Bundle

Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.

BpeTokenizer

Byte-level BPE, the tokenizer of RoBERTa, DistilRoBERTa and GPT-2, as Hugging Face's tokenizers library runs it: the GPT-2 split of the text into words, numbers, punctuation runs and whitespace; each piece's UTF-8 bytes as characters; the merges applied lowest rank first; then RoBERTa's specials, text or a b .

Every encoding records each token's character offsets in the text (GetOffsets), with the leading space a token carries trimmed off when trim_offsets is set, as RoBERTa's are. Special tokens the vocabulary lists, such as , are matched in the text before it is split; one marked lstrip takes the whitespace before it.

Usually built from a model's tokenizer.json by System.NLP.Onnx; the constructor takes the vocabulary and merges themselves.

Implements: SubwordTokenizer

Example

tokenizer := BpeTokenizer->New(tokens, merges, false, true);
encoding := tokenizer->Encode("Hello world", 8);
encoding->GetTokens()->Size()->PrintLine();   # 8: <s> Hello Ġworld </s> <pad> <pad> <pad> <pad>

Operations

AddSpecialToken #

Adds a special token matched in the text before it is split, e.g. ; the specials SetSpecialTokens names are matched already. Adding one again replaces its rules.

method : public : AddSpecialToken(content:String, lstrip:Bool, rstrip:Bool) ~ Bool

Parameters

NameTypeDescription
contentStringthe token, as the vocabulary lists it
lstripBoolwhether it takes the whitespace before it
rstripBoolwhether it takes the whitespace after it

Return

TypeDescription
Booltrue when the vocabulary holds it

Decode #

Turns ids back into text: the tokens' bytes, decoded as UTF-8, special tokens left out.

method : public : Decode(ids:Int[]) ~ String

Parameters

NameTypeDescription
idsInttoken ids

Return

TypeDescription
Stringthe text

Encode #

Encodes one text: text , then to max_len. Text past max_len - 2 tokens is dropped, and the encoding says so.

method : public : Encode(text:String, max_len:Int) ~ TokenEncoding

Parameters

NameTypeDescription
textStringthe text
max_lenIntthe length to truncate and pad to, at least 2

Return

TypeDescription
TokenEncodingthe encoding, or Nil when the tokenizer is not loaded or max_len is below 2

EncodePair #

Encodes a pair: a b , then to max_len; token types are all 0, as RoBERTa's are. When the pair is too long, a token at a time is dropped from the end of whichever text is longer (the second on a tie), Hugging Face's longest_first truncation.

method : public : EncodePair(text_a:String, text_b:String, max_len:Int) ~ TokenEncoding

Parameters

NameTypeDescription
text_aStringthe first text
text_bStringthe second text
max_lenIntthe length to truncate and pad to, at least 4

Return

TypeDescription
TokenEncodingthe encoding, or Nil when the tokenizer is not loaded or max_len is below 4

GetError #

Why the vocabulary did not load, or why the last call returned Nil.

method : public : GetError() ~ String

Return

TypeDescription
Stringthe reason, or Nil

GetId #

A token's id.

method : public : GetId(token:String) ~ Int

Parameters

NameTypeDescription
tokenStringthe token, e.g. "Ġworld"

Return

TypeDescription
Intits id, or -1 when the vocabulary lacks it

GetToken #

The token an id stands for.

method : public : GetToken(id:Int) ~ String

Parameters

NameTypeDescription
idIntthe id

Return

TypeDescription
Stringthe token, or Nil

GetVocabularySize #

The vocabulary's size.

method : public : GetVocabularySize() ~ Int

Return

TypeDescription
Inttokens

IsLoaded #

Whether the vocabulary and specials loaded.

method : public : IsLoaded() ~ Bool

Return

TypeDescription
Booltrue when the tokenizer can encode

New # constructor

Builds a tokenizer from a vocabulary and its merges, with RoBERTa's specials: , , and .

New(tokens:String[], merges:String[], add_prefix_space:Bool, trim_offsets:Bool)

Parameters

NameTypeDescription
tokensStringthe vocabulary, each token's index its id (byte-level strings, e.g. "Ġworld")
mergesStringthe merges in rank order, each "left right"
add_prefix_spaceBoolwhether a space is put before text that does not start with one
trim_offsetsBoolwhether a token's offsets leave out the space it carries

SetSpecialTokens #

Sets the special tokens, for a vocabulary that does not use RoBERTa's names.

method : public : SetSpecialTokens(cls:String, sep:String, pad:String, unk:String) ~ Bool

Parameters

NameTypeDescription
clsStringthe token that starts a text, e.g.
sepStringthe token that ends one, e.g.
padStringthe padding token, e.g.
unkStringthe token for what the vocabulary lacks, e.g. ; may be Nil

Return

TypeDescription
Booltrue when the vocabulary holds them

Tokenize #

The pieces a text becomes, before special tokens are added: byte-level strings, a leading Ġ for a space.

method : public : Tokenize(text:String) ~ String[]

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Stringthe tokens; Nil when the tokenizer is not loaded