Bundle
Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.
SubwordTokenizer
A tokenizer that turns text into a model's input: WordPieceTokenizer (BERT, DistilBERT, MiniLM) and BpeTokenizer (RoBERTa, GPT-2). The task classes of System.NLP.Onnx take either.
Operations
Encode # virtual
Encodes one text, truncated and padded to max_len, special tokens included.
method : public : virtual : Encode(text:String, max_len:Int) ~ TokenEncodingParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
| max_len | Int | the length to truncate and pad to |
Return
| Type | Description |
|---|---|
| TokenEncoding | the encoding, or Nil (see GetError) |
EncodePair # virtual
Encodes a pair of texts, truncated and padded to max_len, special tokens included.
method : public : virtual : EncodePair(text_a:String, text_b:String, max_len:Int) ~ TokenEncodingParameters
| Name | Type | Description |
|---|---|---|
| text_a | String | the first text |
| text_b | String | the second text |
| max_len | Int | the length to truncate and pad to |
Return
| Type | Description |
|---|---|
| TokenEncoding | the encoding, or Nil (see GetError) |