Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.
TokenDictionary
Token to id map with document and collection frequencies, equivalent to gensim.corpora.Dictionary: AddDocuments counts, FilterExtremes drops tokens that are too rare or too common (Gensim's filter_extremes, ids are compacted afterwards in their old order), ToCorpus turns token lists into a bag-of-words corpus. Tokens that are not in the dictionary are ignored by ToCorpus, as Gensim's doc2bow does.
Example
tok := TopicTokenizer->New(TopicStopwords->English());
docs := tok->TokenizeAll(texts);
dict := TokenDictionary->New();
dict->AddDocuments(docs);
report := dict->FilterExtremes(5, 0.5, 100000);
report->ToString()->PrintLine();
corpus := dict->ToCorpus(docs);Operations
- AddDocument
- AddDocuments
- FilterExtremes
- GetCollFreq
- GetDocFreq
- GetError
- GetId
- GetNumDocs
- GetNumTokens
- GetToken
- GetTokens
- Size
- ToCorpus
AddDocument #
Counts one document: every token gets an id (new tokens take the next one), its collection frequency goes up by one per occurrence and its document frequency by one for the document. A Nil document counts as an empty one.
method : public : AddDocument(tokens:Vector<String>) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| tokens | Vector<String> | the document's tokens |
AddDocuments #
Counts many documents.
method : public : AddDocuments(docs:Vector<Vector<String>>) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| docs | Vector<Vector<String>> | token vectors |
FilterExtremes #
Drops tokens like Gensim's filter_extremes(no_below, no_above, keep_n): a token stays when it occurs in at least no_below documents and in at most int(no_above * documents) of them; if more than keep_n remain, the keep_n with the highest document frequency stay (ties go to the smaller id). Ids are then compacted, keeping the old order. With no_below 1, no_above 1.0 and keep_n 0 nothing is removed.
method : public : FilterExtremes(no_below:Int, no_above:Float, keep_n:Int) ~ FilterReportParameters
| Name | Type | Description |
|---|---|---|
| no_below | Int | keep tokens in at least this many documents, 0 or more |
| no_above | Float | keep tokens in at most this fraction of the documents, above 0 and at most 1 |
| keep_n | Int | keep at most this many tokens, or 0 for no limit |
Return
| Type | Description |
|---|---|
| FilterReport | what was removed, or Nil (nothing changed, GetError says why) for a bad argument |
GetCollFreq #
How often a token occurs.
method : public : GetCollFreq(id:Int) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| id | Int | the id |
Return
| Type | Description |
|---|---|
| Int | collection frequency, or -1 when out of range |
GetDocFreq #
In how many documents a token occurs.
method : public : GetDocFreq(id:Int) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| id | Int | the id |
Return
| Type | Description |
|---|---|
| Int | document frequency, or -1 when out of range |
GetError #
Why the last refused call was refused, or "".
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | message |
GetId #
The id of a token.
method : public : GetId(token:String) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| token | String | the token |
Return
| Type | Description |
|---|---|
| Int | id, or -1 when unknown |
GetNumDocs #
Documents counted (a filter does not change this, as in Gensim).
method : public : GetNumDocs() ~ IntReturn
| Type | Description |
|---|---|
| Int | count |
GetNumTokens #
Token occurrences counted and still in the dictionary.
method : public : GetNumTokens() ~ IntReturn
| Type | Description |
|---|---|
| Int | count |
GetToken #
The token with an id.
method : public : GetToken(id:Int) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| id | Int | the id |
Return
| Type | Description |
|---|---|
| String | the token, or Nil when out of range |
GetTokens #
All tokens by id (a copy), the vocabulary an Lda model labels its topics with.
method : public : GetTokens() ~ String[]Return
| Type | Description |
|---|---|
| String | tokens |
Size #
Number of distinct tokens: the dictionary size.
method : public : Size() ~ IntReturn
| Type | Description |
|---|---|
| Int | size |
ToCorpus #
Turns token lists into a bag-of-words corpus over this dictionary; tokens the dictionary does not know are dropped, and a document left with no words stays in the corpus as an empty one.
method : public : ToCorpus(docs:Vector<Vector<String>>) ~ TopicCorpusParameters
| Name | Type | Description |
|---|---|---|
| docs | Vector<Vector<String>> | token vectors, a Nil document counting as empty |
Return
| Type | Description |
|---|---|
| TopicCorpus | the corpus, or Nil when docs is Nil |