v2026.11.2
All Bundles
Bundle

Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.

TopicCorpus

A bag-of-words corpus: for each document the ids of its distinct words and how often each occurs (Gensim's list of (id, count) pairs, stored flat for speed). Build one with TokenDictionary.ToCorpus, or from arrays with Build.

Operations

Build # function

Builds a corpus from flat arrays: document d owns entries offsets[d] to offsets[d+1]-1 of ids and counts.

function : Build(offsets:Int[], ids:Int[], counts:Float[], vocab_size:Int) ~ TopicCorpus

Parameters

NameTypeDescription
offsetsIntone more entry than documents, starting at 0 and not decreasing
idsIntword ids, each in 0 to vocab_size-1 and distinct inside a document
countsFloathow often each word occurs, 1 or more
vocab_sizeIntsize of the vocabulary

Return

TypeDescription
TopicCorpusthe corpus, or Nil when the arrays are Nil, inconsistent or hold an id or count out of range

GetCounts #

The count array itself, not a copy. Do not change it.

method : public : GetCounts() ~ Float[]

Return

TypeDescription
Floatcounts

GetDocFreqs #

In how many documents each word occurs.

method : public : GetDocFreqs() ~ Int[]

Return

TypeDescription
Intone count per word id

GetDocLength #

Word occurrences in one document.

method : public : GetDocLength(doc:Int) ~ Float

Parameters

NameTypeDescription
docIntdocument index

Return

TypeDescription
Floatcount, or -1 for an index out of range

GetIds #

The word-id array itself, not a copy. Do not change it.

method : public : GetIds() ~ Int[]

Return

TypeDescription
Intids

GetNumDocs #

Number of documents, empty ones included.

method : public : GetNumDocs() ~ Int

Return

TypeDescription
Intcount

GetNumUnique #

Distinct words in one document.

method : public : GetNumUnique(doc:Int) ~ Int

Parameters

NameTypeDescription
docIntdocument index

Return

TypeDescription
Intcount, or -1 for an index out of range

GetNumWords #

Total word occurrences in the corpus.

method : public : GetNumWords() ~ Float

Return

TypeDescription
Floatcount

GetOffsets #

The offsets array itself, not a copy: for code that loops over the corpus fast. Do not change it.

method : public : GetOffsets() ~ Int[]

Return

TypeDescription
Intoffsets

GetVocabSize #

Vocabulary size the ids refer to.

method : public : GetVocabSize() ~ Int

Return

TypeDescription
Intsize

GetWordCounts #

Word counts of one document (a copy), in the order of GetWordIds.

method : public : GetWordCounts(doc:Int) ~ Float[]

Parameters

NameTypeDescription
docIntdocument index

Return

TypeDescription
Floatcounts, or Nil for an index out of range

GetWordIds #

Word ids of one document (a copy).

method : public : GetWordIds(doc:Int) ~ Int[]

Parameters

NameTypeDescription
docIntdocument index

Return

TypeDescription
Intids, or Nil for an index out of range

GetWordTotals #

How often each word occurs in the whole corpus.

method : public : GetWordTotals() ~ Float[]

Return

TypeDescription
Floatone count per word id