Objeck Documentation

All Bundles

Bundle Natural language processing toolkit. Provides tokenization, text preprocessing, TF-IDF vectorization, cosine similarity, and sentiment analysis. Works with plain strings — no external model required. Compile with -lib nlp.

Tokenizer

Text tokenization utilities for breaking text into words, sentences, and n-grams

Example

text := "Hello, world! This is a test.";
words := System.NLP.Tokenizer->WordTokenize(text);
sentences := System.NLP.Tokenizer->SentenceTokenize(text);
bigrams := System.NLP.Tokenizer->WordNGrams(words, 2);

CharNGrams # function

Generates character-level n-grams from text

function : CharNGrams(text:String, n:Int) ~ String[]

Parameters

Name	Type	Description
text	String	input text
n	Int	size of n-grams

Return

Type	Description
String	array of n-gram strings

Example

trigrams := Tokenizer->CharNGrams("hello", 3);
each(g in trigrams) {
  g->PrintLine();
};

SentenceTokenize # function

Tokenizes text into sentences using common sentence delimiters

function : SentenceTokenize(text:String) ~ String[]

Parameters

Name	Type	Description
text	String	input text to tokenize

Return

Type	Description
String	array of sentence tokens

Example

sents := Tokenizer->SentenceTokenize("Hello! How are you? I am fine.");
each(s in sents) {
  s->PrintLine();
};

WordNGrams # function

Generates word-level n-grams from tokenized text

function : WordNGrams(tokens:String[], n:Int) ~ String[]

Parameters

Name	Type	Description
tokens	String	array of word tokens
n	Int	size of n-grams

Return

Type	Description
String	array of n-gram strings (words joined by spaces)

Example

words := Tokenizer->WordTokenize("the quick brown fox");
bigrams := Tokenizer->WordNGrams(words, 2);
each(g in bigrams) {
  g->PrintLine();
};

WordTokenize # function

Tokenizes text into words using whitespace and punctuation as delimiters

function : WordTokenize(text:String) ~ String[]

Parameters

Name	Type	Description
text	String	input text to tokenize

Return

Type	Description
String	array of word tokens

Example

words := Tokenizer->WordTokenize("Hello, world! How are you?");
each(w in words) {
  w->PrintLine();
};