v2026.11.2
All Bundles
Bundle

Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.

TextFeatures

Small stylometric features of a text, the kind fake-review and fake-news detectors add to word counts: length, word length, vocabulary richness, punctuation, capitals and sentence length. Plain text in, numbers out; no model.

Definitions (words are Tokenizer->WordTokenize tokens, sentences Tokenizer->SentenceTokenize): characters is the text's length; tokens its word count; average word length the letters per token; type-token ratio the distinct lowercased tokens over tokens; punctuation rate the ASCII punctuation characters over characters; uppercase ratio the capital letters over letters; sentences the sentence count; average sentence length the tokens per sentence. A text with nothing to divide by gives 0.0. A Nil text gives -1 or -1.0 (Extract gives Nil).

Example

text := "Great product! Works perfectly. HIGHLY recommended!!!";
features := System.NLP.TextFeatures->Extract(text);
names := System.NLP.TextFeatures->Names();
each(i : names) {
  "{$(names[i])}: {$(features[i])}"->PrintLine();
};

Operations

AverageSentenceLength # function

The word tokens per sentence.

function : AverageSentenceLength(text:String) ~ Float

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Floatthe mean; 0.0 with no sentences; -1.0 for Nil

AverageWordLength # function

The mean length of the word tokens in characters.

function : AverageWordLength(text:String) ~ Float

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Floatthe mean; 0.0 with no tokens; -1.0 for Nil

CharacterCount # function

The length of a text in characters.

function : CharacterCount(text:String) ~ Int

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Intthe length; -1 for Nil

Extract # function

All the features of a text, in the order of Names.

function : Extract(text:String) ~ Float[]

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Floatthe feature values; Nil for a Nil text

ExtractAll # function

All the features of many texts, a row each: the matrix a classifier takes.

function : ExtractAll(texts:Vector<String>) ~ Float[,]

Parameters

NameTypeDescription
textsVector<String>the texts

Return

TypeDescription
Float[texts, 8]; Nil when texts or an entry is Nil

Example

X := TextFeatures->ExtractAll(reviews);

ExtractAll # function

All the features of many texts, a row each, from an array.

function : ExtractAll(texts:String[]) ~ Float[,]

Parameters

NameTypeDescription
textsStringthe texts

Return

TypeDescription
Float[texts, 8]; Nil when texts or an entry is Nil

ExtractAllWith # function

ExtractWith for many texts, a row each.

function : ExtractAllWith(checker:SpellChecker, texts:Vector<String>) ~ Float[,]

Parameters

NameTypeDescription
checkerSpellCheckera loaded SpellChecker
textsVector<String>the texts

Return

TypeDescription
Float[texts, 10]; Nil when the checker is not loaded, texts or an entry is Nil

ExtractAllWith # function

ExtractWith for many texts, from an array.

function : ExtractAllWith(checker:SpellChecker, texts:String[]) ~ Float[,]

Parameters

NameTypeDescription
checkerSpellCheckera loaded SpellChecker
textsStringthe texts

Return

TypeDescription
Float[texts, 10]; Nil when the checker is not loaded, texts or an entry is Nil

ExtractWith # function

The eight features of Extract followed by the count and the rate of misspelled tokens from a SpellChecker: the feature vector of a fake-review study in one call. Extract is unchanged.

function : ExtractWith(checker:SpellChecker, text:String) ~ Float[]

Parameters

NameTypeDescription
checkerSpellCheckera loaded SpellChecker
textStringthe text

Return

TypeDescription
Floatten values in the order of NamesWith; Nil for a Nil text, a Nil checker or one that is not loaded, never a -1 entry

Example

checker := SpellChecker->New("words.txt");
f := TextFeatures->ExtractWith(checker, "teh product is gud");

Names # function

The names of the values Extract returns, in order: characters, tokens, avg_word_length, type_token_ratio, punctuation_rate, uppercase_ratio, sentences, avg_sentence_length.

function : Names() ~ String[]

Return

TypeDescription
Stringthe feature names

NamesWith # function

The names of the values ExtractWith returns: the eight of Names, then misspelled_count and misspelling_rate.

function : NamesWith() ~ String[]

Return

TypeDescription
Stringten feature names

PunctuationRate # function

The ASCII punctuation characters over all characters.

function : PunctuationRate(text:String) ~ Float

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Floatthe rate; 0.0 for an empty text; -1.0 for Nil

SentenceCount # function

The number of sentences in a text.

function : SentenceCount(text:String) ~ Int

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Intthe count; -1 for Nil

TokenCount # function

The number of word tokens in a text.

function : TokenCount(text:String) ~ Int

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Intthe count; -1 for Nil

TypeTokenRatio # function

The distinct lowercased tokens over all tokens: 1.0 when no word repeats.

function : TypeTokenRatio(text:String) ~ Float

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Floatthe ratio; 0.0 with no tokens; -1.0 for Nil

UppercaseRatio # function

The capital letters over all letters.

function : UppercaseRatio(text:String) ~ Float

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Floatthe ratio; 0.0 with no letters; -1.0 for Nil