Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.
ApiSequenceTokenizer
A tokenizer for sequences of API calls, as malware-analysis datasets record them: CreateFileW,ReadFile,CloseHandle or the same separated by spaces or semicolons.
Fit builds a vocabulary from training data: [PAD]=0, [UNK]=1, [CLS]=2, [SEP]=3, then each API name by descending frequency, ties by name, so the same data always gives the same ids. Encode gives a TokenEncoding like WordPieceTokenizer's: [CLS], the first max_len - 2 calls, [SEP], then [PAD]. Names are case-sensitive unless asked otherwise.
Its ids are for a model trained on them. A pretrained BERT knows only its own WordPiece vocabulary, so for BERT pass the calls as text instead: ToText normalises the separators and truncates by call count, and WordPieceTokenizer splits "CreateFileW" into its pieces.
Example
apis := ApiSequenceTokenizer->New();
apis->Fit(["CreateFileW,ReadFile,CloseHandle", "RegOpenKeyExA RegSetValueExA"]);
encoding := apis->Encode("CreateFileW;WriteFile", 8); # [CLS] CreateFileW [UNK] [SEP] [PAD]...
text := apis->ToText("CreateFileW,ReadFile", 0); # "CreateFileW ReadFile"Operations
Decode #
Turns ids back into API names separated by spaces, leaving out the special tokens.
method : public : Decode(ids:Int[]) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| ids | Int | token ids |
Return
| Type | Description |
|---|---|
| String | the names |
Encode #
Encodes a sequence: [CLS], the first max_len - 2 calls, [SEP], then [PAD] to exactly max_len. A name not in the vocabulary is [UNK].
method : public : Encode(sequence:String, max_len:Int) ~ TokenEncodingParameters
| Name | Type | Description |
|---|---|---|
| sequence | String | the calls |
| max_len | Int | the length to truncate and pad to, at least 2 |
Return
| Type | Description |
|---|---|
| TokenEncoding | the encoding, or Nil when max_len is below 2 |
EncodeBatch #
Encodes several sequences, each with Encode.
method : public : EncodeBatch(sequences:String[], max_len:Int) ~ TokenEncoding[]Parameters
| Name | Type | Description |
|---|---|---|
| sequences | String | the sequences |
| max_len | Int | the length to truncate and pad to |
Return
| Type | Description |
|---|---|
| TokenEncoding | one encoding per sequence, or Nil when max_len is below 2 |
Fit #
Builds the vocabulary from training sequences, replacing any earlier one.
method : public : Fit(sequences:String[]) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| sequences | String | training sequences |
Return
| Type | Description |
|---|---|
| Int | the vocabulary size, special tokens included |
GetVocabSize #
Number of tokens in the vocabulary, special tokens included.
method : public : GetVocabSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | vocabulary size |
IdToToken #
The name with an id.
method : public : IdToToken(id:Int) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| id | Int | token id |
Return
| Type | Description |
|---|---|
| String | the name, or Nil when the id is out of range |
Load # function
Reads a vocabulary Save wrote.
function : Load(path:String, lower_case:Bool) ~ ApiSequenceTokenizerParameters
| Name | Type | Description |
|---|---|---|
| path | String | the file |
| lower_case | Bool | fold names to lowercase, as the tokenizer that saved it did |
Return
| Type | Description |
|---|---|
| ApiSequenceTokenizer | the tokenizer, or Nil when the file cannot be read or does not start with [PAD], [UNK], [CLS], [SEP] |
New # constructor
An empty tokenizer.
New(lower_case:Bool, min_count:Int)Parameters
| Name | Type | Description |
|---|---|---|
| lower_case | Bool | fold names to lowercase |
| min_count | Int | least number of training occurrences for a name to get an id; rarer names encode as [UNK] |
Save #
Writes the vocabulary, one token per line, the line number its id: the vocab.txt layout.
method : public : Save(path:String) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| path | String | the file |
Return
| Type | Description |
|---|---|
| Bool | true when written |
Split #
Splits a sequence into API names at commas, semicolons and whitespace, dropping empty names, and lowercasing them if configured.
method : public : Split(sequence:String) ~ String[]Parameters
| Name | Type | Description |
|---|---|---|
| sequence | String | the calls |
Return
| Type | Description |
|---|---|
| String | the names |