v2026.11.2
All Bundles
Bundle

Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.

TopicTokenizer

Splits tweets and other short texts into lowercase tokens for topic modeling: runs of the letters a-z and digits 0-9 (any other byte, including accented and non-Latin characters, ends a token, so this is meant for English text), so "don't" gives "don" and "t". It drops URLs (a word that has "://" or starts with "www."),

Operations

New # constructor

A tokenizer that keeps every token of 2 or more characters.

New()

New # constructor

A tokenizer that also drops the words of a stopword list.

New(stop:TopicStopwords)

Parameters

NameTypeDescription
stopTopicStopwordsthe list, or Nil for none

SetDropNumbers #

Chooses whether tokens made only of digits are dropped (the default).

method : public : SetDropNumbers(drop:Bool) ~ Nil

Parameters

NameTypeDescription
dropBooltrue to drop them

SetLengths #

Sets the shortest and longest token kept.

method : public : SetLengths(min:Int, max:Int) ~ Bool

Parameters

NameTypeDescription
minIntshortest length, 1 or more
maxIntlongest length, at least min

Return

TypeDescription
Booltrue if accepted, false (nothing changed) otherwise

Tokenize #

Splits one text. A Nil or empty text gives an empty vector.

method : public : Tokenize(text:String) ~ Vector<String>

Parameters

NameTypeDescription
textStringthe text

Return

TypeDescription
Vector<String>tokens in order

TokenizeAll #

Splits many texts; a Nil text gives an empty document.

method : public : TokenizeAll(texts:Vector<String>) ~ Vector<Vector<String>>

Parameters

NameTypeDescription
textsVector<String>the texts

Return

TypeDescription
Vector<Vector<String>>one token vector per text, or Nil when texts is Nil