v2026.11.2
All Bundles
Bundle

Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.

ApiSequenceTokenizer

A tokenizer for sequences of API calls, as malware-analysis datasets record them: CreateFileW,ReadFile,CloseHandle or the same separated by spaces or semicolons.

Fit builds a vocabulary from training data: [PAD]=0, [UNK]=1, [CLS]=2, [SEP]=3, then each API name by descending frequency, ties by name, so the same data always gives the same ids. Encode gives a TokenEncoding like WordPieceTokenizer's: [CLS], the first max_len - 2 calls, [SEP], then [PAD]. Names are case-sensitive unless asked otherwise.

Its ids are for a model trained on them. A pretrained BERT knows only its own WordPiece vocabulary, so for BERT pass the calls as text instead: ToText normalises the separators and truncates by call count, and WordPieceTokenizer splits "CreateFileW" into its pieces.

Example

apis := ApiSequenceTokenizer->New();
apis->Fit(["CreateFileW,ReadFile,CloseHandle", "RegOpenKeyExA RegSetValueExA"]);
encoding := apis->Encode("CreateFileW;WriteFile", 8);   # [CLS] CreateFileW [UNK] [SEP] [PAD]...
text := apis->ToText("CreateFileW,ReadFile", 0);        # "CreateFileW ReadFile"

Operations

Decode #

Turns ids back into API names separated by spaces, leaving out the special tokens.

method : public : Decode(ids:Int[]) ~ String

Parameters

NameTypeDescription
idsInttoken ids

Return

TypeDescription
Stringthe names

Encode #

Encodes a sequence: [CLS], the first max_len - 2 calls, [SEP], then [PAD] to exactly max_len. A name not in the vocabulary is [UNK].

method : public : Encode(sequence:String, max_len:Int) ~ TokenEncoding

Parameters

NameTypeDescription
sequenceStringthe calls
max_lenIntthe length to truncate and pad to, at least 2

Return

TypeDescription
TokenEncodingthe encoding, or Nil when max_len is below 2

EncodeBatch #

Encodes several sequences, each with Encode.

method : public : EncodeBatch(sequences:String[], max_len:Int) ~ TokenEncoding[]

Parameters

NameTypeDescription
sequencesStringthe sequences
max_lenIntthe length to truncate and pad to

Return

TypeDescription
TokenEncodingone encoding per sequence, or Nil when max_len is below 2

Fit #

Builds the vocabulary from training sequences, replacing any earlier one.

method : public : Fit(sequences:String[]) ~ Int

Parameters

NameTypeDescription
sequencesStringtraining sequences

Return

TypeDescription
Intthe vocabulary size, special tokens included

GetVocabSize #

Number of tokens in the vocabulary, special tokens included.

method : public : GetVocabSize() ~ Int

Return

TypeDescription
Intvocabulary size

IdToToken #

The name with an id.

method : public : IdToToken(id:Int) ~ String

Parameters

NameTypeDescription
idInttoken id

Return

TypeDescription
Stringthe name, or Nil when the id is out of range

Load # function

Reads a vocabulary Save wrote.

function : Load(path:String, lower_case:Bool) ~ ApiSequenceTokenizer

Parameters

NameTypeDescription
pathStringthe file
lower_caseBoolfold names to lowercase, as the tokenizer that saved it did

Return

TypeDescription
ApiSequenceTokenizerthe tokenizer, or Nil when the file cannot be read or does not start with [PAD], [UNK], [CLS], [SEP]

New # constructor

An empty tokenizer, case-sensitive, keeping every name seen at least once.

New()

New # constructor

An empty tokenizer.

New(lower_case:Bool, min_count:Int)

Parameters

NameTypeDescription
lower_caseBoolfold names to lowercase
min_countIntleast number of training occurrences for a name to get an id; rarer names encode as [UNK]

Save #

Writes the vocabulary, one token per line, the line number its id: the vocab.txt layout.

method : public : Save(path:String) ~ Bool

Parameters

NameTypeDescription
pathStringthe file

Return

TypeDescription
Booltrue when written

Split #

Splits a sequence into API names at commas, semicolons and whitespace, dropping empty names, and lowercasing them if configured.

method : public : Split(sequence:String) ~ String[]

Parameters

NameTypeDescription
sequenceStringthe calls

Return

TypeDescription
Stringthe names

ToText #

The calls as text for a WordPiece tokenizer: names separated by single spaces.

method : public : ToText(sequence:String, max_calls:Int) ~ String

Parameters

NameTypeDescription
sequenceStringthe calls
max_callsIntkeep at most this many calls; 0 or less keeps all

Return

TypeDescription
Stringthe text

TokenToId #

A name's id.

method : public : TokenToId(token:String) ~ Int

Parameters

NameTypeDescription
tokenStringan API name or special token

Return

TypeDescription
Intits id, or -1 when the vocabulary lacks it