Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.
WordPieceTokenizer
BERT's tokenizer: a BasicTokenizer followed by WordPiece, reading a model's vocab.txt, and the encoding a BERT-family model takes as input.
The BasicTokenizer drops control characters, puts spaces around CJK ideographs, splits on whitespace, then per word lowercases (an option), strips accents (an option: decompose, then drop combining marks) and splits off every punctuation character. WordPiece then splits each word greedily, longest match first, into vocabulary pieces, continuations prefixed "##"; a word with a remainder no piece matches, or longer than 100 characters, becomes one [UNK]. The special tokens [CLS], [SEP], [PAD], [MASK] and [UNK] are never split when they appear in the text.
This is Hugging Face's BertTokenizer (the Python one), checked against a transcription of it on a corpus (programs/regression/nlp_wordpiece_parity.obs). Its Unicode tables are generated from Python's unicodedata (tools/nlp/gen_wordpiece_tables.py). Two differences: Hugging Face composes the text to NFC first, which matters only to a model that does not strip accents given decomposed text; and it drops code points Unicode has not assigned, which this keeps.
Defaults are bert-base-uncased's: lowercase and strip accents. A cased model needs both off.
Example
tokenizer := WordPieceTokenizer->New("vocab.txt");
pieces := tokenizer->Tokenize("He's unaffable!"); # he ' s un ##aff ##able !
encoding := tokenizer->Encode("He's unaffable!", 16);
ids := encoding->GetIds(); # [CLS] ... [SEP] then [PAD] to 16
pair := tokenizer->EncodePair("a question", "a passage", 32);
tokenizer->Decode(ids)->PrintLine(); # he ' s unaffable !Operations
- New
- BasicTokenize
- Decode
- Encode
- EncodeBatch
- EncodePair
- GetClsId
- GetError
- GetMaskId
- GetPadId
- GetSepId
- GetUnkId
- GetVocabSize
- IdToToken
- IsLoaded
- IsLowerCase
- IsStripAccents
- SetMaxWordChars
- TokenToId
- Tokenize
- TokenizeToIds
BasicTokenize #
Splits text into words and punctuation, BERT's BasicTokenizer: no vocabulary is used.
method : public : BasicTokenize(text:String) ~ String[]Parameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| String | words, lowercased and accent-stripped as configured |
Example
words := tokenizer->BasicTokenize("Hello, World!"); # hello , world !Decode #
Turns ids back into text: pieces joined with spaces, a "##" continuation joined to the piece before it, special tokens left out. Lowercasing and stripped accents do not come back.
method : public : Decode(ids:Int[]) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| ids | Int | token ids |
Return
| Type | Description |
|---|---|
| String | the text |
Decode #
Turns ids back into text.
method : public : Decode(ids:Int[], skip_special:Bool) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| ids | Int | token ids |
| skip_special | Bool | leave out [CLS], [SEP], [PAD], [MASK] and [UNK] |
Return
| Type | Description |
|---|---|
| String | the text |
Encode #
Encodes one text: [CLS], its pieces, [SEP], then [PAD] to exactly max_len. A text too long keeps its beginning.
method : public : Encode(text:String, max_len:Int) ~ TokenEncodingParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
| max_len | Int | the length to truncate and pad to, at least 2 |
Return
| Type | Description |
|---|---|
| TokenEncoding | the encoding, or Nil when the tokenizer is not loaded or max_len is below 2 |
Example
encoding := tokenizer->Encode("the quick brown fox", 128);EncodeBatch #
Encodes several texts, each with Encode.
method : public : EncodeBatch(texts:String[], max_len:Int) ~ TokenEncoding[]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
| max_len | Int | the length to truncate and pad to |
Return
| Type | Description |
|---|---|
| TokenEncoding | one encoding per text, or Nil when any cannot be encoded |
EncodePair #
Encodes a pair: [CLS] a [SEP] b [SEP], token types 0 through the first [SEP] and 1 after, then [PAD] to max_len. When the pair is too long, a token at a time is dropped from the end of whichever text is longer (the second on a tie), Hugging Face's longest_first truncation.
method : public : EncodePair(text_a:String, text_b:String, max_len:Int) ~ TokenEncodingParameters
| Name | Type | Description |
|---|---|---|
| text_a | String | the first text |
| text_b | String | the second text |
| max_len | Int | the length to truncate and pad to, at least 3 |
Return
| Type | Description |
|---|---|
| TokenEncoding | the encoding, or Nil when the tokenizer is not loaded or max_len is below 3 |
GetError #
Why the tokenizer is not loaded, or why the last Encode returned Nil.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or Nil |
GetMaskId #
Id of [MASK].
method : public : GetMaskId() ~ IntReturn
| Type | Description |
|---|---|
| Int | id, -1 when absent |
GetVocabSize #
Number of tokens in the vocabulary.
method : public : GetVocabSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | vocabulary size |
IdToToken #
The token with an id.
method : public : IdToToken(id:Int) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| id | Int | token id |
Return
| Type | Description |
|---|---|
| String | the token, or Nil when the id is out of range |
IsLoaded #
Whether the vocabulary loaded and names [UNK], [CLS], [SEP] and [PAD].
method : public : IsLoaded() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when the tokenizer can encode |
IsLowerCase #
Whether text is lowercased.
method : public : IsLowerCase() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true for an uncased tokenizer |
IsStripAccents #
Whether accents are stripped.
method : public : IsStripAccents() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when accents are removed |
New # constructor
Loads a vocabulary for an uncased model: lowercase and strip accents.
New(vocab_path:String)Parameters
| Name | Type | Description |
|---|---|---|
| vocab_path | String | vocab.txt, one token per line, the line number its id |
New # constructor
Loads a vocabulary.
New(vocab_path:String, lower_case:Bool, strip_accents:Bool)Parameters
| Name | Type | Description |
|---|---|---|
| vocab_path | String | vocab.txt, one token per line, the line number its id |
| lower_case | Bool | lowercase the text (true for an uncased model) |
| strip_accents | Bool | remove accents (true for an uncased model) |
New # constructor
Builds a tokenizer from a vocabulary held in memory.
New(tokens:String[], lower_case:Bool, strip_accents:Bool)Parameters
| Name | Type | Description |
|---|---|---|
| tokens | String | the vocabulary, each token's index its id |
| lower_case | Bool | lowercase the text |
| strip_accents | Bool | remove accents |
SetMaxWordChars #
Sets the longest word, in characters, WordPiece will split; a longer one becomes [UNK]. BERT's value is 100.
method : public : SetMaxWordChars(max_word_chars:Int) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| max_word_chars | Int | longest word |
TokenToId #
A token's id.
method : public : TokenToId(token:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| token | String | the token, e.g. "##able" |
Return
| Type | Description |
|---|---|
| Int | its id, or -1 when the vocabulary lacks it |