Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.
TokenEncoding
One text encoded for a BERT-family model: the token ids, the attention mask and the token type (segment) ids, all the same length, ready to become a model's input_ids, attention_mask and token_type_ids. Produced by WordPieceTokenizer and ApiSequenceTokenizer.
Example
tokenizer := WordPieceTokenizer->New("vocab.txt");
encoding := tokenizer->Encode("unaffable", 8);
encoding->GetTokens()->Size()->PrintLine(); # 8: [CLS] un ##aff ##able [SEP] [PAD] [PAD] [PAD]
encoding->GetLength()->PrintLine(); # 5: the tokens before the paddingOperations
GetAttentionMask #
Attention mask: 1 for a real token, 0 for padding.
method : public : GetAttentionMask() ~ Int[]Return
| Type | Description |
|---|---|
| Int | mask |
GetLength #
Tokens before the padding, [CLS] and [SEP] included.
method : public : GetLength() ~ IntReturn
| Type | Description |
|---|---|
| Int | unpadded length |
GetOffsets #
Each token's character offsets in its text, as Hugging Face's offset_mapping: the token at position i spans [offsets[2i], offsets[2i + 1]) of the text it came from; special tokens and padding are (0, 0).
method : public : GetOffsets() ~ Int[]Return
| Type | Description |
|---|---|
| Int | two per id, or Nil when the tokenizer does not record offsets |
GetSize #
Total length, padding included: the max_len it was encoded to.
method : public : GetSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | padded length |
GetTokenTypeIds #
Token type (segment) ids: 0 for the first text, 1 for the second of a pair.
method : public : GetTokenTypeIds() ~ Int[]Return
| Type | Description |
|---|---|
| Int | token type ids |
GetTokens #
The token strings, one per id.
method : public : GetTokens() ~ String[]Return
| Type | Description |
|---|---|
| String | tokens |
IsTruncated #
Whether text was dropped to fit max_len.
method : public : IsTruncated() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when truncated |
New # constructor
Constructor, for a tokenizer of the program's own.
New(ids:Int[], attention_mask:Int[], token_type_ids:Int[], tokens:String[], length:Int, truncated:Bool)Parameters
| Name | Type | Description |
|---|---|---|
| ids | Int | token ids, padding included |
| attention_mask | Int | 1 for a real token, 0 for padding |
| token_type_ids | Int | segment of each token, 0 or 1 |
| tokens | String | the token strings, one per id |
| length | Int | tokens before the padding, special tokens included |
| truncated | Bool | whether text was dropped to fit |
New # constructor
Constructor with each token's character offsets in the text.
New(ids:Int[], attention_mask:Int[], token_type_ids:Int[], tokens:String[], length:Int, truncated:Bool, offsets:Int[])Parameters
| Name | Type | Description |
|---|---|---|
| ids | Int | token ids, padding included |
| attention_mask | Int | 1 for a real token, 0 for padding |
| token_type_ids | Int | segment of each token, 0 or 1 |
| tokens | String | the token strings, one per id |
| length | Int | tokens before the padding, special tokens included |
| truncated | Bool | whether text was dropped to fit |
| offsets | Int | start and end of each token in its text, two per id: [2i] and [2i + 1] |