v2026.11.2
All Bundles
Bundle

Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.

FilterReport

What FilterExtremes removed: the dictionary size and the share of token occurrences lost, the two numbers the min-frequency experiment reports.

Operations

GetTokensAfter #

Token occurrences left.

method : public : GetTokensAfter() ~ Int

Return

TypeDescription
Intcount

GetTokensBefore #

Token occurrences in the corpus before the filter.

method : public : GetTokensBefore() ~ Int

Return

TypeDescription
Intcount

GetTokensRemovedPercent #

Percent of the token OCCURRENCES (not the distinct tokens) that were removed, 0 to 100; 0 when there were none.

method : public : GetTokensRemovedPercent() ~ Float

Return

TypeDescription
Floatpercent

GetTypesAfter #

Distinct tokens left: the dictionary size.

method : public : GetTypesAfter() ~ Int

Return

TypeDescription
Intcount

GetTypesBefore #

Distinct tokens before the filter.

method : public : GetTypesBefore() ~ Int

Return

TypeDescription
Intcount

GetTypesRemovedPercent #

Percent of the distinct tokens that were removed, 0 to 100; 0 when there were none.

method : public : GetTypesRemovedPercent() ~ Float

Return

TypeDescription
Floatpercent