Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.
TextFeatures
Small stylometric features of a text, the kind fake-review and fake-news detectors add to word counts: length, word length, vocabulary richness, punctuation, capitals and sentence length. Plain text in, numbers out; no model.
Definitions (words are Tokenizer->WordTokenize tokens, sentences Tokenizer->SentenceTokenize): characters is the text's length; tokens its word count; average word length the letters per token; type-token ratio the distinct lowercased tokens over tokens; punctuation rate the ASCII punctuation characters over characters; uppercase ratio the capital letters over letters; sentences the sentence count; average sentence length the tokens per sentence. A text with nothing to divide by gives 0.0. A Nil text gives -1 or -1.0 (Extract gives Nil).
Example
text := "Great product! Works perfectly. HIGHLY recommended!!!";
features := System.NLP.TextFeatures->Extract(text);
names := System.NLP.TextFeatures->Names();
each(i : names) {
"{$(names[i])}: {$(features[i])}"->PrintLine();
};Operations
- AverageSentenceLength
- AverageWordLength
- CharacterCount
- Extract
- ExtractAll
- ExtractAllWith
- ExtractWith
- Names
- NamesWith
- PunctuationRate
- SentenceCount
- TokenCount
- TypeTokenRatio
- UppercaseRatio
AverageSentenceLength # function
The word tokens per sentence.
function : AverageSentenceLength(text:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Float | the mean; 0.0 with no sentences; -1.0 for Nil |
AverageWordLength # function
The mean length of the word tokens in characters.
function : AverageWordLength(text:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Float | the mean; 0.0 with no tokens; -1.0 for Nil |
CharacterCount # function
The length of a text in characters.
function : CharacterCount(text:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Int | the length; -1 for Nil |
Extract # function
All the features of a text, in the order of Names.
function : Extract(text:String) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Float | the feature values; Nil for a Nil text |
ExtractAll # function
All the features of many texts, a row each: the matrix a classifier takes.
function : ExtractAll(texts:Vector<String>) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| texts | Vector<String> | the texts |
Return
| Type | Description |
|---|---|
| Float | [texts, 8]; Nil when texts or an entry is Nil |
Example
X := TextFeatures->ExtractAll(reviews);ExtractAll # function
All the features of many texts, a row each, from an array.
function : ExtractAll(texts:String[]) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| texts | String | the texts |
Return
| Type | Description |
|---|---|
| Float | [texts, 8]; Nil when texts or an entry is Nil |
ExtractAllWith # function
ExtractWith for many texts, a row each.
function : ExtractAllWith(checker:SpellChecker, texts:Vector<String>) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| checker | SpellChecker | a loaded SpellChecker |
| texts | Vector<String> | the texts |
Return
| Type | Description |
|---|---|
| Float | [texts, 10]; Nil when the checker is not loaded, texts or an entry is Nil |
ExtractAllWith # function
ExtractWith for many texts, from an array.
function : ExtractAllWith(checker:SpellChecker, texts:String[]) ~ Float[,]Parameters
| Name | Type | Description |
|---|---|---|
| checker | SpellChecker | a loaded SpellChecker |
| texts | String | the texts |
Return
| Type | Description |
|---|---|
| Float | [texts, 10]; Nil when the checker is not loaded, texts or an entry is Nil |
ExtractWith # function
The eight features of Extract followed by the count and the rate of misspelled tokens from a SpellChecker: the feature vector of a fake-review study in one call. Extract is unchanged.
function : ExtractWith(checker:SpellChecker, text:String) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| checker | SpellChecker | a loaded SpellChecker |
| text | String | the text |
Return
| Type | Description |
|---|---|
| Float | ten values in the order of NamesWith; Nil for a Nil text, a Nil checker or one that is not loaded, never a -1 entry |
Example
checker := SpellChecker->New("words.txt");
f := TextFeatures->ExtractWith(checker, "teh product is gud");Names # function
The names of the values Extract returns, in order: characters, tokens, avg_word_length, type_token_ratio, punctuation_rate, uppercase_ratio, sentences, avg_sentence_length.
function : Names() ~ String[]Return
| Type | Description |
|---|---|
| String | the feature names |
NamesWith # function
The names of the values ExtractWith returns: the eight of Names, then misspelled_count and misspelling_rate.
function : NamesWith() ~ String[]Return
| Type | Description |
|---|---|
| String | ten feature names |
PunctuationRate # function
The ASCII punctuation characters over all characters.
function : PunctuationRate(text:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Float | the rate; 0.0 for an empty text; -1.0 for Nil |
SentenceCount # function
The number of sentences in a text.
function : SentenceCount(text:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Int | the count; -1 for Nil |
TokenCount # function
The number of word tokens in a text.
function : TokenCount(text:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Int | the count; -1 for Nil |