Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.
TopicStopwords
Stopword lists as data, for the three strategies of the lab: none, standard English, and English plus a custom list. A list is a set of lowercase words; Union combines lists without changing either.
Operations
Add #
Adds a word (lowercased). Nil and empty words are ignored.
method : public : Add(word:String) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| word | String | the word |
Cyber # function
An OPTIONAL starting point for the "English plus custom cyber stoplist" strategy; it is small on purpose and is not a researched list. It holds generic Twitter noise (rt, via, amp, http, https, www, com, co, ly, bit, ...) and a few words that occur in nearly every cybersecurity tweet and so carry no topic ("cyber", "cybersecurity", "infosec", "security"). Edit it to taste: build your own with FromArray, or add words with Add.
function : Cyber() ~ TopicStopwordsReturn
| Type | Description |
|---|---|
| TopicStopwords | the list |
English # function
The standard English list: the 179 words of NLTK's stopwords corpus for English (nltk.corpus.stopwords.words("english"), as shipped with NLTK 3.x), which is a public-domain word list reproduced here so no download is needed. It contains contractions such as "don't" and their stems ("don", "t"); the tokenizer in this library splits on apostrophes, so the stems are what match.
function : English() ~ TopicStopwordsReturn
| Type | Description |
|---|---|
| TopicStopwords | the list |
FromArray # function
A list of your own words (lowercased on the way in); Nil words are ignored.
function : FromArray(words:String[]) ~ TopicStopwordsParameters
| Name | Type | Description |
|---|---|---|
| words | String | the words |
Return
| Type | Description |
|---|---|
| TopicStopwords | the list |
Has #
Tells whether a word is on the list; the word is compared as given, so pass lowercase.
method : public : Has(word:String) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| word | String | the word |
Return
| Type | Description |
|---|---|
| Bool | true when listed |
ToArray #
The words, in no particular order.
method : public : ToArray() ~ String[]Return
| Type | Description |
|---|---|
| String | a new array |
Union #
A new list with the words of both; either may be Nil.
method : public : Union(extra:TopicStopwords) ~ TopicStopwordsParameters
| Name | Type | Description |
|---|---|---|
| extra | TopicStopwords | another list |
Return
| Type | Description |
|---|---|
| TopicStopwords | the combined list |