Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.
TopicTokenizer
Splits tweets and other short texts into lowercase tokens for topic modeling: runs of the letters a-z and digits 0-9 (any other byte, including accented and non-Latin characters, ends a token, so this is meant for English text), so "don't" gives "don" and "t". It drops URLs (a word that has "://" or starts with "www."),
Operations
New # constructor
A tokenizer that also drops the words of a stopword list.
New(stop:TopicStopwords)Parameters
| Name | Type | Description |
|---|---|---|
| stop | TopicStopwords | the list, or Nil for none |
SetDropNumbers #
Chooses whether tokens made only of digits are dropped (the default).
method : public : SetDropNumbers(drop:Bool) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| drop | Bool | true to drop them |
SetLengths #
Sets the shortest and longest token kept.
method : public : SetLengths(min:Int, max:Int) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| min | Int | shortest length, 1 or more |
| max | Int | longest length, at least min |
Return
| Type | Description |
|---|---|
| Bool | true if accepted, false (nothing changed) otherwise |