v2026.11.2
All Bundles
Bundle

Topic modeling for text (System.NLP.Topics): the pieces that turn a MongoDB BSON dump or a list of strings into the bag-of-words corpus that the LDA model (nlp_topics_lda.obs) trains on, with the same steps as a Gensim workflow: decode, tokenize, remove stopwords, build a dictionary, filter infrequent tokens, build the corpus. Compile with -lib nlp_topics,json,plot,gen_collect.

BsonReader

Decodes MongoDB BSON (bsonspec.org version 1.1) into Data.JSON elements. A .bson dump from mongodump is a plain concatenation of documents, each starting with its own little-endian int32 total length; HasNext/Next walk it one document at a time without decoding the rest.

Nothing here ends the program on bad input. A truncated file, a length that runs past the data or past its parent document, a length below the 5-byte minimum, a missing terminator, an unknown or unsupported type byte, an unterminated name or string, a Boolean that is not 0 or 1 and nesting deeper than 100 levels (MongoDB's own limit) all make Next return Nil and set GetError() to a message with the byte offset; after an error HasNext is false, so a loop always ends. Every length is checked against the bytes that are left before it is used, so a hostile file cannot make the reader read out of range or loop without advancing.

How BSON types map onto JSON elements. Documents become objects and arrays become arrays; strings, int32 and int64 become strings and numbers; a double becomes a number, except NaN and the infinities, which become the strings "NaN", "Infinity" and "-Infinity" (JSON has no such numbers); true and false and null map to themselves, and the deprecated undefined is null. An ObjectId becomes its 24-character lowercase hex string. A UTC datetime becomes an ISO-8601 string with milliseconds ("2021-03-04T05:06:07.089Z"), or "ms:" and the number for a date outside years 0000 to 9999. Binary becomes {"$binary": hex, "$type": subtype}, a timestamp {"$timestamp": {"t": seconds, "i": increment}}, a regular expression {"$regex": pattern, "$options": flags}, JavaScript code and symbols plain strings, min and max keys {"$minKey": 1} and {"$maxKey": 1}. Decimal128 is NOT decoded: it becomes {"$decimal128": hex} holding its 16 raw bytes as stored (little-endian), so the value is kept and a program can tell it apart. DBPointer and code-with-scope are refused with an error. Strings are bytes as stored, UTF-8 is not validated; a repeated name in one document keeps the last value. Int64 values beyond what a JSON number text round-trips are still exact here, because numbers are carried as text.

Example

reader := BsonReader->FromFile("cybertweets.bson");
tweets := reader->ExtractStrings("text");
"{$tweets->Size()} tweets"->PrintLine();

Operations

ExtractStrings #

Takes one string field from every remaining document, for example the tweet text. The field may be a dotted path into embedded documents ("user.description"). Documents without the field, or where it is not a string, are skipped and counted in GetSkipped().

method : public : ExtractStrings(field:String) ~ Vector<String>

Parameters

NameTypeDescription
fieldStringfield name or dotted path

Return

TypeDescription
Vector<String>the strings in file order, or Nil when the field name is empty or a document is malformed (see GetError; the strings read before the error are dropped)

FromFile # function

Reads a whole .bson file into memory and makes a reader over it.

function : FromFile(path:String) ~ BsonReader

Parameters

NameTypeDescription
pathStringthe file

Return

TypeDescription
BsonReaderthe reader, or Nil when the file is missing or unreadable

GetError #

Why the last failure happened, or an empty string.

method : public : GetError() ~ String

Return

TypeDescription
Stringmessage with the byte offset, or ""

GetPosition #

Byte offset of the next document.

method : public : GetPosition() ~ Int

Return

TypeDescription
Intoffset

GetSkipped #

Number of documents ExtractStrings left out because the field was missing or not a string.

method : public : GetSkipped() ~ Int

Return

TypeDescription
Intcount from the last ExtractStrings call

HasNext #

True while there is another document and no error has happened.

method : public : HasNext() ~ Bool

Return

TypeDescription
Booltrue if Next may be called

New # constructor

Creates a reader over bytes already in memory. Nil bytes give a reader with an error and nothing to read.

New(data:Byte[])

Parameters

NameTypeDescription
dataBytethe BSON bytes (not copied; do not change them while reading)

Next #

Decodes the next document.

method : public : Next() ~ JsonElement

Return

TypeDescription
JsonElementthe document as a JSON object, or Nil at the end of the data or on an error (see GetError)

ReadAll #

Decodes every remaining document. Unlike Next this is all or nothing: one bad document gives Nil.

method : public : ReadAll() ~ Vector<JsonElement>

Return

TypeDescription
Vector<JsonElement>the documents, or Nil on an error (see GetError)

ReadStrings # function

One call for the common case: read a .bson file and take one string field from every document.

function : ReadStrings(path:String, field:String) ~ Vector<String>

Parameters

NameTypeDescription
pathStringthe .bson file
fieldStringfield name or dotted path

Return

TypeDescription
Vector<String>the strings, or Nil when the file is unreadable or malformed