v2026.11.2
All Bundles
Bundle

Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.

TokenEncoding

One text encoded for a BERT-family model: the token ids, the attention mask and the token type (segment) ids, all the same length, ready to become a model's input_ids, attention_mask and token_type_ids. Produced by WordPieceTokenizer and ApiSequenceTokenizer.

Example

tokenizer := WordPieceTokenizer->New("vocab.txt");
encoding := tokenizer->Encode("unaffable", 8);
encoding->GetTokens()->Size()->PrintLine();   # 8: [CLS] un ##aff ##able [SEP] [PAD] [PAD] [PAD]
encoding->GetLength()->PrintLine();           # 5: the tokens before the padding

Operations

GetAttentionMask #

Attention mask: 1 for a real token, 0 for padding.

method : public : GetAttentionMask() ~ Int[]

Return

TypeDescription
Intmask

GetIds #

Token ids, padding included.

method : public : GetIds() ~ Int[]

Return

TypeDescription
Intids

GetLength #

Tokens before the padding, [CLS] and [SEP] included.

method : public : GetLength() ~ Int

Return

TypeDescription
Intunpadded length

GetOffsets #

Each token's character offsets in its text, as Hugging Face's offset_mapping: the token at position i spans [offsets[2i], offsets[2i + 1]) of the text it came from; special tokens and padding are (0, 0).

method : public : GetOffsets() ~ Int[]

Return

TypeDescription
Inttwo per id, or Nil when the tokenizer does not record offsets

GetSize #

Total length, padding included: the max_len it was encoded to.

method : public : GetSize() ~ Int

Return

TypeDescription
Intpadded length

GetTokenTypeIds #

Token type (segment) ids: 0 for the first text, 1 for the second of a pair.

method : public : GetTokenTypeIds() ~ Int[]

Return

TypeDescription
Inttoken type ids

GetTokens #

The token strings, one per id.

method : public : GetTokens() ~ String[]

Return

TypeDescription
Stringtokens

IsTruncated #

Whether text was dropped to fit max_len.

method : public : IsTruncated() ~ Bool

Return

TypeDescription
Booltrue when truncated

New # constructor

Constructor, for a tokenizer of the program's own.

New(ids:Int[], attention_mask:Int[], token_type_ids:Int[], tokens:String[], length:Int, truncated:Bool)

Parameters

NameTypeDescription
idsInttoken ids, padding included
attention_maskInt1 for a real token, 0 for padding
token_type_idsIntsegment of each token, 0 or 1
tokensStringthe token strings, one per id
lengthInttokens before the padding, special tokens included
truncatedBoolwhether text was dropped to fit

New # constructor

Constructor with each token's character offsets in the text.

New(ids:Int[], attention_mask:Int[], token_type_ids:Int[], tokens:String[], length:Int, truncated:Bool, offsets:Int[])

Parameters

NameTypeDescription
idsInttoken ids, padding included
attention_maskInt1 for a real token, 0 for padding
token_type_idsIntsegment of each token, 0 or 1
tokensStringthe token strings, one per id
lengthInttokens before the padding, special tokens included
truncatedBoolwhether text was dropped to fit
offsetsIntstart and end of each token in its text, two per id: [2i] and [2i + 1]