v2026.11.2
All Bundles
Bundle

Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.

TopicStopwords

Stopword lists as data, for the three strategies of the lab: none, standard English, and English plus a custom list. A list is a set of lowercase words; Union combines lists without changing either.

Operations

Add #

Adds a word (lowercased). Nil and empty words are ignored.

method : public : Add(word:String) ~ Nil

Parameters

NameTypeDescription
wordStringthe word

Cyber # function

An OPTIONAL starting point for the "English plus custom cyber stoplist" strategy; it is small on purpose and is not a researched list. It holds generic Twitter noise (rt, via, amp, http, https, www, com, co, ly, bit, ...) and a few words that occur in nearly every cybersecurity tweet and so carry no topic ("cyber", "cybersecurity", "infosec", "security"). Edit it to taste: build your own with FromArray, or add words with Add.

function : Cyber() ~ TopicStopwords

Return

TypeDescription
TopicStopwordsthe list

English # function

The standard English list: the 179 words of NLTK's stopwords corpus for English (nltk.corpus.stopwords.words("english"), as shipped with NLTK 3.x), which is a public-domain word list reproduced here so no download is needed. It contains contractions such as "don't" and their stems ("don", "t"); the tokenizer in this library splits on apostrophes, so the stems are what match.

function : English() ~ TopicStopwords

Return

TypeDescription
TopicStopwordsthe list

FromArray # function

A list of your own words (lowercased on the way in); Nil words are ignored.

function : FromArray(words:String[]) ~ TopicStopwords

Parameters

NameTypeDescription
wordsStringthe words

Return

TypeDescription
TopicStopwordsthe list

Has #

Tells whether a word is on the list; the word is compared as given, so pass lowercase.

method : public : Has(word:String) ~ Bool

Parameters

NameTypeDescription
wordStringthe word

Return

TypeDescription
Booltrue when listed

New # constructor

An empty list: the strategy "no stopword removal".

New()

Size #

Number of words.

method : public : Size() ~ Int

Return

TypeDescription
Intsize

ToArray #

The words, in no particular order.

method : public : ToArray() ~ String[]

Return

TypeDescription
Stringa new array

Union #

A new list with the words of both; either may be Nil.

method : public : Union(extra:TopicStopwords) ~ TopicStopwords

Parameters

NameTypeDescription
extraTopicStopwordsanother list

Return

TypeDescription
TopicStopwordsthe combined list