Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.
FilterReport
What FilterExtremes removed: the dictionary size and the share of token occurrences lost, the two numbers the min-frequency experiment reports.
Operations
- GetTokensAfter
- GetTokensBefore
- GetTokensRemovedPercent
- GetTypesAfter
- GetTypesBefore
- GetTypesRemovedPercent
GetTokensAfter #
Token occurrences left.
method : public : GetTokensAfter() ~ IntReturn
| Type | Description |
|---|---|
| Int | count |
GetTokensBefore #
Token occurrences in the corpus before the filter.
method : public : GetTokensBefore() ~ IntReturn
| Type | Description |
|---|---|
| Int | count |
GetTokensRemovedPercent #
Percent of the token OCCURRENCES (not the distinct tokens) that were removed, 0 to 100; 0 when there were none.
method : public : GetTokensRemovedPercent() ~ FloatReturn
| Type | Description |
|---|---|
| Float | percent |
GetTypesAfter #
Distinct tokens left: the dictionary size.
method : public : GetTypesAfter() ~ IntReturn
| Type | Description |
|---|---|
| Int | count |