Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.
BsonReader
Decodes MongoDB BSON (bsonspec.org version 1.1) into Data.JSON elements. A .bson dump from mongodump is a plain concatenation of documents, each starting with its own little-endian int32 total length; HasNext/Next walk it one document at a time without decoding the rest.
Nothing here ends the program on bad input. A truncated file, a length that runs past the data or past its parent document, a length below the 5-byte minimum, a missing terminator, an unknown or unsupported type byte, an unterminated name or string, a Boolean that is not 0 or 1 and nesting deeper than 100 levels (MongoDB's own limit) all make Next return Nil and set GetError() to a message with the byte offset; after an error HasNext is false, so a loop always ends. Every length is checked against the bytes that are left before it is used, so a hostile file cannot make the reader read out of range or loop without advancing.
How BSON types map onto JSON elements. Documents become objects and arrays become arrays; strings, int32 and int64 become strings and numbers; a double becomes a number, except NaN and the infinities, which become the strings "NaN", "Infinity" and "-Infinity" (JSON has no such numbers); true and false and null map to themselves, and the deprecated undefined is null. An ObjectId becomes its 24-character lowercase hex string. A UTC datetime becomes an ISO-8601 string with milliseconds ("2021-03-04T05:06:07.089Z"), or "ms:" and the number for a date outside years 0000 to 9999. Binary becomes {"$binary": hex, "$type": subtype}, a timestamp {"$timestamp": {"t": seconds, "i": increment}}, a regular expression {"$regex": pattern, "$options": flags}, JavaScript code and symbols plain strings, min and max keys {"$minKey": 1} and {"$maxKey": 1}. Decimal128 is NOT decoded: it becomes {"$decimal128": hex} holding its 16 raw bytes as stored (little-endian), so the value is kept and a program can tell it apart. DBPointer and code-with-scope are refused with an error. Strings are bytes as stored, UTF-8 is not validated; a repeated name in one document keeps the last value. Int64 values beyond what a JSON number text round-trips are still exact here, because numbers are carried as text.
Example
reader := BsonReader->FromFile("cybertweets.bson");
tweets := reader->ExtractStrings("text");
"{$tweets->Size()} tweets"->PrintLine();Operations
ExtractStrings #
Takes one string field from every remaining document, for example the tweet text. The field may be a dotted path into embedded documents ("user.description"). Documents without the field, or where it is not a string, are skipped and counted in GetSkipped().
method : public : ExtractStrings(field:String) ~ Vector<String>Parameters
| Name | Type | Description |
|---|---|---|
| field | String | field name or dotted path |
Return
| Type | Description |
|---|---|
| Vector<String> | the strings in file order, or Nil when the field name is empty or a document is malformed (see GetError; the strings read before the error are dropped) |
FromFile # function
Reads a whole .bson file into memory and makes a reader over it.
function : FromFile(path:String) ~ BsonReaderParameters
| Name | Type | Description |
|---|---|---|
| path | String | the file |
Return
| Type | Description |
|---|---|
| BsonReader | the reader, or Nil when the file is missing or unreadable |
GetError #
Why the last failure happened, or an empty string.
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | message with the byte offset, or "" |
GetPosition #
Byte offset of the next document.
method : public : GetPosition() ~ IntReturn
| Type | Description |
|---|---|
| Int | offset |
GetSkipped #
Number of documents ExtractStrings left out because the field was missing or not a string.
method : public : GetSkipped() ~ IntReturn
| Type | Description |
|---|---|
| Int | count from the last ExtractStrings call |
HasNext #
True while there is another document and no error has happened.
method : public : HasNext() ~ BoolReturn
| Type | Description |
|---|---|
| Bool | true if Next may be called |
New # constructor
Creates a reader over bytes already in memory. Nil bytes give a reader with an error and nothing to read.
New(data:Byte[])Parameters
| Name | Type | Description |
|---|---|---|
| data | Byte | the BSON bytes (not copied; do not change them while reading) |
Next #
Decodes the next document.
method : public : Next() ~ JsonElementReturn
| Type | Description |
|---|---|
| JsonElement | the document as a JSON object, or Nil at the end of the data or on an error (see GetError) |
ReadAll #
Decodes every remaining document. Unlike Next this is all or nothing: one bad document gives Nil.
method : public : ReadAll() ~ Vector<JsonElement>Return
| Type | Description |
|---|---|
| Vector<JsonElement> | the documents, or Nil on an error (see GetError) |
ReadStrings # function
One call for the common case: read a .bson file and take one string field from every document.
function : ReadStrings(path:String, field:String) ~ Vector<String>Parameters
| Name | Type | Description |
|---|---|---|
| path | String | the .bson file |
| field | String | field name or dotted path |
Return
| Type | Description |
|---|---|
| Vector<String> | the strings, or Nil when the file is unreadable or malformed |