Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.
BpeTokenizer
Byte-level BPE, the tokenizer of RoBERTa, DistilRoBERTa and GPT-2, as Hugging Face's tokenizers library runs it: the GPT-2 split of the text into words, numbers, punctuation runs and whitespace; each piece's UTF-8 bytes as characters; the merges applied lowest rank first; then RoBERTa's specials, text or a b .
Every encoding records each token's character offsets in the text (GetOffsets), with the leading space a token carries trimmed off when trim_offsets is set, as RoBERTa's are. Special tokens the vocabulary lists, such as
Usually built from a model's tokenizer.json by System.NLP.Onnx; the constructor takes the vocabulary and merges themselves.
Example
tokenizer := BpeTokenizer->New(tokens, merges, false, true);
encoding := tokenizer->Encode("Hello world", 8);
encoding->GetTokens()->Size()->PrintLine(); # 8: <s> Hello Ġworld </s> <pad> <pad> <pad> <pad>Operations
- New
- AddSpecialToken
- Decode
- Encode
- EncodePair
- GetError
- GetId
- GetToken
- GetVocabularySize
- IsLoaded
- SetSpecialTokens
- Tokenize
AddSpecialToken #
Adds a special token matched in the text before it is split, e.g.
method : public : AddSpecialToken(content:String, lstrip:Bool, rstrip:Bool) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| content | String | the token, as the vocabulary lists it |
| lstrip | Bool | whether it takes the whitespace before it |
| rstrip | Bool | whether it takes the whitespace after it |
Return
| Type | Description |
|---|---|
| Bool | true when the vocabulary holds it |
Decode #
Turns ids back into text: the tokens' bytes, decoded as UTF-8, special tokens left out.
method : public : Decode(ids:Int[]) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| ids | Int | token ids |
Return
| Type | Description |
|---|---|
| String | the text |
Encode #
Encodes one text: text , then
method : public : Encode(text:String, max_len:Int) ~ TokenEncodingParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
| max_len | Int | the length to truncate and pad to, at least 2 |
Return
| Type | Description |
|---|---|
| TokenEncoding | the encoding, or Nil when the tokenizer is not loaded or max_len is below 2 |
EncodePair #
Encodes a pair: a b , then
method : public : EncodePair(text_a:String, text_b:String, max_len:Int) ~ TokenEncodingParameters
| Name | Type | Description |
|---|---|---|
| text_a | String | the first text |
| text_b | String | the second text |
| max_len | Int | the length to truncate and pad to, at least 4 |
Return
| Type | Description |
|---|---|
| TokenEncoding | the encoding, or Nil when the tokenizer is not loaded or max_len is below 4 |
GetError #
Why the vocabulary did not load, or why the last call returned Nil.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason, or Nil |
GetId #
A token's id.
method : public : GetId(token:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| token | String | the token, e.g. "Ġworld" |
Return
| Type | Description |
|---|---|
| Int | its id, or -1 when the vocabulary lacks it |
GetToken #
The token an id stands for.
method : public : GetToken(id:Int) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| id | Int | the id |
Return
| Type | Description |
|---|---|
| String | the token, or Nil |
GetVocabularySize #
The vocabulary's size.
method : public : GetVocabularySize() ~ IntReturn
| Type | Description |
|---|---|
| Int | tokens |
IsLoaded #
Whether the vocabulary and specials loaded.
method : public : IsLoaded() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when the tokenizer can encode |
New # constructor
Builds a tokenizer from a vocabulary and its merges, with RoBERTa's specials: , ,
New(tokens:String[], merges:String[], add_prefix_space:Bool, trim_offsets:Bool)Parameters
| Name | Type | Description |
|---|---|---|
| tokens | String | the vocabulary, each token's index its id (byte-level strings, e.g. "Ġworld") |
| merges | String | the merges in rank order, each "left right" |
| add_prefix_space | Bool | whether a space is put before text that does not start with one |
| trim_offsets | Bool | whether a token's offsets leave out the space it carries |
SetSpecialTokens #
Sets the special tokens, for a vocabulary that does not use RoBERTa's names.
method : public : SetSpecialTokens(cls:String, sep:String, pad:String, unk:String) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| cls | String | the token that starts a text, e.g. |
| sep | String | the token that ends one, e.g. |
| pad | String | the padding token, e.g. |
| unk | String | the token for what the vocabulary lacks, e.g. |
Return
| Type | Description |
|---|---|
| Bool | true when the vocabulary holds them |