v2026.11.2
All Bundles
Bundle

Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.

TokenDictionary

Token to id map with document and collection frequencies, equivalent to gensim.corpora.Dictionary: AddDocuments counts, FilterExtremes drops tokens that are too rare or too common (Gensim's filter_extremes, ids are compacted afterwards in their old order), ToCorpus turns token lists into a bag-of-words corpus. Tokens that are not in the dictionary are ignored by ToCorpus, as Gensim's doc2bow does.

Example

tok := TopicTokenizer->New(TopicStopwords->English());
docs := tok->TokenizeAll(texts);
dict := TokenDictionary->New();
dict->AddDocuments(docs);
report := dict->FilterExtremes(5, 0.5, 100000);
report->ToString()->PrintLine();
corpus := dict->ToCorpus(docs);

Operations

AddDocument #

Counts one document: every token gets an id (new tokens take the next one), its collection frequency goes up by one per occurrence and its document frequency by one for the document. A Nil document counts as an empty one.

method : public : AddDocument(tokens:Vector<String>) ~ Nil

Parameters

NameTypeDescription
tokensVector<String>the document's tokens

AddDocuments #

Counts many documents.

method : public : AddDocuments(docs:Vector<Vector<String>>) ~ Nil

Parameters

NameTypeDescription
docsVector<Vector<String>>token vectors

FilterExtremes #

Drops tokens like Gensim's filter_extremes(no_below, no_above, keep_n): a token stays when it occurs in at least no_below documents and in at most int(no_above * documents) of them; if more than keep_n remain, the keep_n with the highest document frequency stay (ties go to the smaller id). Ids are then compacted, keeping the old order. With no_below 1, no_above 1.0 and keep_n 0 nothing is removed.

method : public : FilterExtremes(no_below:Int, no_above:Float, keep_n:Int) ~ FilterReport

Parameters

NameTypeDescription
no_belowIntkeep tokens in at least this many documents, 0 or more
no_aboveFloatkeep tokens in at most this fraction of the documents, above 0 and at most 1
keep_nIntkeep at most this many tokens, or 0 for no limit

Return

TypeDescription
FilterReportwhat was removed, or Nil (nothing changed, GetError says why) for a bad argument

GetCollFreq #

How often a token occurs.

method : public : GetCollFreq(id:Int) ~ Int

Parameters

NameTypeDescription
idIntthe id

Return

TypeDescription
Intcollection frequency, or -1 when out of range

GetDocFreq #

In how many documents a token occurs.

method : public : GetDocFreq(id:Int) ~ Int

Parameters

NameTypeDescription
idIntthe id

Return

TypeDescription
Intdocument frequency, or -1 when out of range

GetError #

Why the last refused call was refused, or "".

method : public : GetError() ~ String

Return

TypeDescription
Stringmessage

GetId #

The id of a token.

method : public : GetId(token:String) ~ Int

Parameters

NameTypeDescription
tokenStringthe token

Return

TypeDescription
Intid, or -1 when unknown

GetNumDocs #

Documents counted (a filter does not change this, as in Gensim).

method : public : GetNumDocs() ~ Int

Return

TypeDescription
Intcount

GetNumTokens #

Token occurrences counted and still in the dictionary.

method : public : GetNumTokens() ~ Int

Return

TypeDescription
Intcount

GetToken #

The token with an id.

method : public : GetToken(id:Int) ~ String

Parameters

NameTypeDescription
idIntthe id

Return

TypeDescription
Stringthe token, or Nil when out of range

GetTokens #

All tokens by id (a copy), the vocabulary an Lda model labels its topics with.

method : public : GetTokens() ~ String[]

Return

TypeDescription
Stringtokens

Size #

Number of distinct tokens: the dictionary size.

method : public : Size() ~ Int

Return

TypeDescription
Intsize

ToCorpus #

Turns token lists into a bag-of-words corpus over this dictionary; tokens the dictionary does not know are dropped, and a document left with no words stays in the corpus as an empty one.

method : public : ToCorpus(docs:Vector<Vector<String>>) ~ TopicCorpus

Parameters

NameTypeDescription
docsVector<Vector<String>>token vectors, a Nil document counting as empty

Return

TypeDescription
TopicCorpusthe corpus, or Nil when docs is Nil