Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.
TopicCorpus
A bag-of-words corpus: for each document the ids of its distinct words and how often each occurs (Gensim's list of (id, count) pairs, stored flat for speed). Build one with TokenDictionary.ToCorpus, or from arrays with Build.
Operations
- Build
- GetCounts
- GetDocFreqs
- GetDocLength
- GetIds
- GetNumDocs
- GetNumUnique
- GetNumWords
- GetOffsets
- GetVocabSize
- GetWordCounts
- GetWordIds
- GetWordTotals
Build # function
Builds a corpus from flat arrays: document d owns entries offsets[d] to offsets[d+1]-1 of ids and counts.
function : Build(offsets:Int[], ids:Int[], counts:Float[], vocab_size:Int) ~ TopicCorpusParameters
| Name | Type | Description |
|---|---|---|
| offsets | Int | one more entry than documents, starting at 0 and not decreasing |
| ids | Int | word ids, each in 0 to vocab_size-1 and distinct inside a document |
| counts | Float | how often each word occurs, 1 or more |
| vocab_size | Int | size of the vocabulary |
Return
| Type | Description |
|---|---|
| TopicCorpus | the corpus, or Nil when the arrays are Nil, inconsistent or hold an id or count out of range |
GetCounts #
The count array itself, not a copy. Do not change it.
method : public : GetCounts() ~ Float[]Return
| Type | Description |
|---|---|
| Float | counts |
GetDocFreqs #
In how many documents each word occurs.
method : public : GetDocFreqs() ~ Int[]Return
| Type | Description |
|---|---|
| Int | one count per word id |
GetDocLength #
Word occurrences in one document.
method : public : GetDocLength(doc:Int) ~ FloatParameters
| Name | Type | Description |
|---|---|---|
| doc | Int | document index |
Return
| Type | Description |
|---|---|
| Float | count, or -1 for an index out of range |
GetIds #
The word-id array itself, not a copy. Do not change it.
method : public : GetIds() ~ Int[]Return
| Type | Description |
|---|---|
| Int | ids |
GetNumDocs #
Number of documents, empty ones included.
method : public : GetNumDocs() ~ IntReturn
| Type | Description |
|---|---|
| Int | count |
GetNumUnique #
Distinct words in one document.
method : public : GetNumUnique(doc:Int) ~ IntParameters
| Name | Type | Description |
|---|---|---|
| doc | Int | document index |
Return
| Type | Description |
|---|---|
| Int | count, or -1 for an index out of range |
GetNumWords #
Total word occurrences in the corpus.
method : public : GetNumWords() ~ FloatReturn
| Type | Description |
|---|---|
| Float | count |
GetOffsets #
The offsets array itself, not a copy: for code that loops over the corpus fast. Do not change it.
method : public : GetOffsets() ~ Int[]Return
| Type | Description |
|---|---|
| Int | offsets |
GetVocabSize #
Vocabulary size the ids refer to.
method : public : GetVocabSize() ~ IntReturn
| Type | Description |
|---|---|
| Int | size |
GetWordCounts #
Word counts of one document (a copy), in the order of GetWordIds.
method : public : GetWordCounts(doc:Int) ~ Float[]Parameters
| Name | Type | Description |
|---|---|---|
| doc | Int | document index |
Return
| Type | Description |
|---|---|
| Float | counts, or Nil for an index out of range |