Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.
SpellChecker
Counts misspelled words against a word list the program supplies, for the misspelling feature of fake-review and fake-news detectors. No dictionary ships with Objeck: load a newline-delimited list of words, one per line, such as a Linux /usr/share/dict/words, the SCOWL / aspell word lists (http://wordlist.aspell.net) or the "words_alpha.txt" list of English words; check the licence of the list you pick. Matching ignores case.
Words are found the way Tokenizer->WordTokenize finds them, so the same text gives the same tokens as TextPreprocessor and TF_IDF. A token that holds a digit ("COVID-19", "3.14") is never counted as misspelled, and a possessive ("dog's") is checked without its "'s". The rate is the misspelled tokens over all tokens, digit tokens included.
A checker that is not loaded, and a Nil text, give -1 (count) or -1.0 (rate) rather than a crash; an empty text has no tokens and gives 0 and 0.0.
Example
checker := System.NLP.SpellChecker->New("words.txt");
if(<>checker->IsLoaded()) {
checker->GetError()->PrintLine();
return;
};
"{$checker->CountMisspelled("teh cat sat on teh mat")} of 6"->PrintLine();
"rate {$checker->MisspellingRate("teh cat sat on teh mat")}"->PrintLine();
suggestions := checker->Suggest("teh", 2, 3);Operations
- New
- AddWord
- Contains
- CountMisspelled
- GetError
- GetWordCount
- IsLoaded
- Load
- Misspelled
- MisspellingRate
- Suggest
- TokenCount
AddWord #
Adds one word to the list, e.g. a domain term.
method : public : AddWord(word:String) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| word | String | the word |
Contains #
Whether a word is in the list, ignoring case; a possessive ("dog's") passes when its stem does.
method : public : Contains(word:String) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| word | String | the word |
Return
| Type | Description |
|---|---|
| Bool | true when the word is known |
CountMisspelled #
The number of misspelled tokens in a text.
method : public : CountMisspelled(text:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Int | the count; -1 when the checker is not loaded or the text is Nil |
GetError #
Why the list did not load.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | the reason; "" when there is nothing to report (never Nil) |
GetWordCount #
The number of distinct words in the list.
method : public : GetWordCount() ~ IntReturn
| Type | Description |
|---|---|
| Int | word count |
IsLoaded #
Whether the list holds words, so counts mean something.
method : public : IsLoaded() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true when loaded |
Load #
Adds the words of a file to the list, case folded.
method : public : Load(path:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| path | String | newline-delimited word list |
Return
| Type | Description |
|---|---|
| Int | the number of words read, or -1 when the file did not open (see GetError) |
Misspelled #
The misspelled tokens of a text, in order, case kept.
method : public : Misspelled(text:String) ~ String[]Parameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| String | the tokens not in the list; Nil when the checker is not loaded or the text is Nil |
MisspellingRate #
Misspelled tokens as a share of all tokens.
method : public : MisspellingRate(text:String) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| text | String | the text |
Return
| Type | Description |
|---|---|
| Float | 0.0 to 1.0; 0.0 for a text with no tokens; -1.0 when the checker is not loaded or the text is Nil |
New # constructor
Constructor that loads a word list.
New(path:String)Parameters
| Name | Type | Description |
|---|---|---|
| path | String | newline-delimited word list; blank lines and lines starting with '#' are skipped |
Suggest #
Words of the list within an edit distance (Levenshtein) of a word, nearest first and alphabetical within a distance. The list is scanned whole, so this is for the few misspellings a report shows, not for every token of a corpus.
method : public : Suggest(word:String, max_distance:Int, max_count:Int) ~ String[]Parameters
| Name | Type | Description |
|---|---|---|
| word | String | the word to correct |
| max_distance | Int | the largest edit distance accepted, at least 1 |
| max_count | Int | the most suggestions to return, at least 1 |
Return
| Type | Description |
|---|---|
| String | the suggestions, empty when none; Nil when the checker is not loaded or the word is Nil |