v2026.11.2
All Bundles
Bundle

Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.

WordPieceTokenizer

BERT's tokenizer: a BasicTokenizer followed by WordPiece, reading a model's vocab.txt, and the encoding a BERT-family model takes as input.

The BasicTokenizer drops control characters, puts spaces around CJK ideographs, splits on whitespace, then per word lowercases (an option), strips accents (an option: decompose, then drop combining marks) and splits off every punctuation character. WordPiece then splits each word greedily, longest match first, into vocabulary pieces, continuations prefixed "##"; a word with a remainder no piece matches, or longer than 100 characters, becomes one [UNK]. The special tokens [CLS], [SEP], [PAD], [MASK] and [UNK] are never split when they appear in the text.

This is Hugging Face's BertTokenizer (the Python one), checked against a transcription of it on a corpus (programs/regression/nlp_wordpiece_parity.obs). Its Unicode tables are generated from Python's unicodedata (tools/nlp/gen_wordpiece_tables.py). Two differences: Hugging Face composes the text to NFC first, which matters only to a model that does not strip accents given decomposed text; and it drops code points Unicode has not assigned, which this keeps.

Defaults are bert-base-uncased's: lowercase and strip accents. A cased model needs both off.

Implements: SubwordTokenizer

Example

tokenizer := WordPieceTokenizer->New("vocab.txt");
pieces := tokenizer->Tokenize("He's unaffable!");    # he ' s un ##aff ##able !
encoding := tokenizer->Encode("He's unaffable!", 16);
ids := encoding->GetIds();                            # [CLS] ... [SEP] then [PAD] to 16
pair := tokenizer->EncodePair("a question", "a passage", 32);
tokenizer->Decode(ids)->PrintLine();                  # he ' s unaffable !

Operations

BasicTokenize #

Splits text into words and punctuation, BERT's BasicTokenizer: no vocabulary is used.

method : public : BasicTokenize(text:String) ~ String[]

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Stringwords, lowercased and accent-stripped as configured

Example

words := tokenizer->BasicTokenize("Hello, World!");   # hello , world !

Decode #

Turns ids back into text: pieces joined with spaces, a "##" continuation joined to the piece before it, special tokens left out. Lowercasing and stripped accents do not come back.

method : public : Decode(ids:Int[]) ~ String

Parameters

NameTypeDescription
idsInttoken ids

Return

TypeDescription
Stringthe text

Decode #

Turns ids back into text.

method : public : Decode(ids:Int[], skip_special:Bool) ~ String

Parameters

NameTypeDescription
idsInttoken ids
skip_specialBoolleave out [CLS], [SEP], [PAD], [MASK] and [UNK]

Return

TypeDescription
Stringthe text

Encode #

Encodes one text: [CLS], its pieces, [SEP], then [PAD] to exactly max_len. A text too long keeps its beginning.

method : public : Encode(text:String, max_len:Int) ~ TokenEncoding

Parameters

NameTypeDescription
textStringthe text
max_lenIntthe length to truncate and pad to, at least 2

Return

TypeDescription
TokenEncodingthe encoding, or Nil when the tokenizer is not loaded or max_len is below 2

Example

encoding := tokenizer->Encode("the quick brown fox", 128);

EncodeBatch #

Encodes several texts, each with Encode.

method : public : EncodeBatch(texts:String[], max_len:Int) ~ TokenEncoding[]

Parameters

NameTypeDescription
textsStringthe texts
max_lenIntthe length to truncate and pad to

Return

TypeDescription
TokenEncodingone encoding per text, or Nil when any cannot be encoded

EncodePair #

Encodes a pair: [CLS] a [SEP] b [SEP], token types 0 through the first [SEP] and 1 after, then [PAD] to max_len. When the pair is too long, a token at a time is dropped from the end of whichever text is longer (the second on a tie), Hugging Face's longest_first truncation.

method : public : EncodePair(text_a:String, text_b:String, max_len:Int) ~ TokenEncoding

Parameters

NameTypeDescription
text_aStringthe first text
text_bStringthe second text
max_lenIntthe length to truncate and pad to, at least 3

Return

TypeDescription
TokenEncodingthe encoding, or Nil when the tokenizer is not loaded or max_len is below 3

GetClsId #

Id of [CLS].

method : public : GetClsId() ~ Int

Return

TypeDescription
Intid, -1 when absent

GetError #

Why the tokenizer is not loaded, or why the last Encode returned Nil.

method : public : GetError() ~ String

Return

TypeDescription
Stringthe reason, or Nil

GetMaskId #

Id of [MASK].

method : public : GetMaskId() ~ Int

Return

TypeDescription
Intid, -1 when absent

GetPadId #

Id of [PAD].

method : public : GetPadId() ~ Int

Return

TypeDescription
Intid, -1 when absent

GetSepId #

Id of [SEP].

method : public : GetSepId() ~ Int

Return

TypeDescription
Intid, -1 when absent

GetUnkId #

Id of [UNK].

method : public : GetUnkId() ~ Int

Return

TypeDescription
Intid, -1 when absent

GetVocabSize #

Number of tokens in the vocabulary.

method : public : GetVocabSize() ~ Int

Return

TypeDescription
Intvocabulary size

IdToToken #

The token with an id.

method : public : IdToToken(id:Int) ~ String

Parameters

NameTypeDescription
idInttoken id

Return

TypeDescription
Stringthe token, or Nil when the id is out of range

IsLoaded #

Whether the vocabulary loaded and names [UNK], [CLS], [SEP] and [PAD].

method : public : IsLoaded() ~ Bool

Return

TypeDescription
Booltrue when the tokenizer can encode

IsLowerCase #

Whether text is lowercased.

method : public : IsLowerCase() ~ Bool

Return

TypeDescription
Booltrue for an uncased tokenizer

IsStripAccents #

Whether accents are stripped.

method : public : IsStripAccents() ~ Bool

Return

TypeDescription
Booltrue when accents are removed

New # constructor

Loads a vocabulary for an uncased model: lowercase and strip accents.

New(vocab_path:String)

Parameters

NameTypeDescription
vocab_pathStringvocab.txt, one token per line, the line number its id

New # constructor

Loads a vocabulary.

New(vocab_path:String, lower_case:Bool, strip_accents:Bool)

Parameters

NameTypeDescription
vocab_pathStringvocab.txt, one token per line, the line number its id
lower_caseBoollowercase the text (true for an uncased model)
strip_accentsBoolremove accents (true for an uncased model)

New # constructor

Builds a tokenizer from a vocabulary held in memory.

New(tokens:String[], lower_case:Bool, strip_accents:Bool)

Parameters

NameTypeDescription
tokensStringthe vocabulary, each token's index its id
lower_caseBoollowercase the text
strip_accentsBoolremove accents

SetMaxWordChars #

Sets the longest word, in characters, WordPiece will split; a longer one becomes [UNK]. BERT's value is 100.

method : public : SetMaxWordChars(max_word_chars:Int) ~ Nil

Parameters

NameTypeDescription
max_word_charsIntlongest word

TokenToId #

A token's id.

method : public : TokenToId(token:String) ~ Int

Parameters

NameTypeDescription
tokenStringthe token, e.g. "##able"

Return

TypeDescription
Intits id, or -1 when the vocabulary lacks it

Tokenize #

Splits text into WordPiece tokens, without [CLS], [SEP] or padding.

method : public : Tokenize(text:String) ~ String[]

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Stringthe pieces

Example

tokenizer->Tokenize("unaffable");   # un ##aff ##able

TokenizeToIds #

Splits text into WordPiece token ids, without [CLS], [SEP] or padding.

method : public : TokenizeToIds(text:String) ~ Int[]

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Intthe ids