v2026.11.2
All Bundles
Bundle

In-memory query engine over typed columns. Tables are filled row by row or loaded from sources (CSV files, folders, the program's own objects) and queried with SQL-like statements; aggregates, Describe() and Finder (file search) are included. Compile with -lib query.

ArchiveSource

A zip or tar archive as tables, without extracting it: a listing of its members as a tree (like FilesystemSource's), and each member in a format the library reads as a table of its own, named by the member's path ('data/runs.csv'; in SQL, "data/runs.csv" in double quotes). Members are read by extension: '.csv', '.tsv', '.jsonl' and '.ndjson', '.json', '.otab', and '.xml' once SetRowElement names the row element, each also with '.gz'. Other members, '.txt' included, are listed only; ReadBytes and ReadText read any of them.

The archive is a '.zip' (stored and deflate members, ZIP64, a self-extracting prefix), a '.tar' (POSIX ustar, pax and GNU long names) or a gzipped tar ('.tar.gz', '.tgz'), told apart by content rather than name, from a file, an https:// URL or bytes in memory.

Untrusted archives are safe to read. Nothing is written to disk. A member whose path is absolute or climbs out with '..' is refused, as are duplicate names in a zip and members whose data overlap. Each zip member is inflated to at most the size its header states and checked against its CRC32, so a zip bomb or a damaged member is an error naming it, never wrong data. Reading stops at 256 MB for one member and 1 GB for the whole archive unless SetLimits says otherwise. Encrypted members, compression other than deflate, and archives split across disks are reported as unsupported. Links are listed but never followed.

The listing table is named after the archive ('results' for 'results.zip'), with columns id, parent_id (empty at the top), depth (0 at the top), name, path, ext, is_dir, type ('file', 'dir', 'symlink', 'hardlink' or 'other'), size, compressed_size and method (zip only; method is 'stored', 'deflate' or the name of another), crc32 (zip only), modified and link (a link's target). Folders that the archive implies but does not list get rows too, so every member has a parent. A tar member repeated later in the archive replaces the earlier one's table, as extracting would.

Implements: Source

Example

catalog := Catalog->New();
if(catalog->Load(ArchiveSource->New("results.zip"))) {
  rs := catalog->Query("select path, size from results where ext = 'csv' order by size desc");
  runs := catalog->Query("select name, avg(score) from \"2026/runs.csv\" group by name");
};

Operations

ColumnNames # function

Gets the listing's column names

function : ColumnNames() ~ String[]

Return

TypeDescription
Stringcolumn names

FromBytes # function

Makes a source over an archive already in memory

function : FromBytes(name:String, bytes:Byte[]) ~ ArchiveSource

Parameters

NameTypeDescription
nameStringthe listing table's name
bytesBytearchive bytes

Return

TypeDescription
ArchiveSourcesource

GetError #

Gets the reason the last read failed

method : public : GetError() ~ String

Return

TypeDescription
Stringerror, Nil if none

GetMembers #

Gets the paths of the archive's files, in archive order

method : public : GetMembers() ~ String[]

Return

TypeDescription
Stringmember paths, Nil if the archive cannot be read (see GetError)

GetTables #

Gets the tables the archive provides, the listing first; the members' columns are known only after reading them

method : public : GetTables() ~ TableSpec[]

Return

TypeDescription
TableSpectable shapes

List #

Lists the archive's members as a table

method : public : List() ~ Table

Return

TypeDescription
Tablelisting, Nil if the archive cannot be read (see GetError)

Load #

Loads the listing and every member readable as a table into a catalog. Nothing is added unless every member reads. The scan hint is not pushed into members.

method : public : Load(catalog:Catalog, hint:ScanHint) ~ Bool

Parameters

NameTypeDescription
catalogCatalogcatalog
hintScanHintignored; the engine applies the query afterwards

Return

TypeDescription
Booltrue if loaded

New # constructor

Constructor; the listing table is named after the archive

New(path:String)

Parameters

NameTypeDescription
pathStringarchive file or URL ('.zip', '.tar', '.tar.gz' or '.tgz')

ReadBytes #

Reads one member's bytes, checked against its CRC32 and limits

method : public : ReadBytes(member:String) ~ Byte[]

Parameters

NameTypeDescription
memberStringmember path, as the listing gives it

Return

TypeDescription
Bytebytes, Nil if missing or unreadable (see GetError)

ReadTable #

Reads one member as a table, by its extension, whether or not it matches the pattern

method : public : ReadTable(member:String) ~ Table

Parameters

NameTypeDescription
memberStringmember path, as the listing gives it

Return

TypeDescription
Tablethe member's table (a JSON or XML member's child tables are left out), Nil if it cannot be read as one (see GetError)

ReadText #

Reads one member as UTF-8 text

method : public : ReadText(member:String) ~ String

Parameters

NameTypeDescription
memberStringmember path, as the listing gives it

Return

TypeDescription
Stringtext, Nil if missing or unreadable (see GetError)

SetLimits #

Sets how much may be decompressed: a member larger than 'member_bytes' is refused, and so is a read that would take the archive's total past 'total_bytes'. The defaults are 256 MB and 1 GB.

method : public : SetLimits(member_bytes:Int, total_bytes:Int) ~ Nil

Parameters

NameTypeDescription
member_bytesIntmost bytes one member may hold uncompressed
total_bytesIntmost bytes all members read together may hold uncompressed

SetName #

Sets the listing table's name

method : public : SetName(name:String) ~ Nil

Parameters

NameTypeDescription
nameStringtable name

SetPattern #

Sets which members become tables: a pattern over member paths, where '*' matches any run of characters (slashes included) and '?' any one. All readable members by default.

method : public : SetPattern(pattern:String) ~ Nil

Parameters

NameTypeDescription
patternStringpattern, such as 'runs/*.csv'

SetRowElement #

Names the element that is a row in XML members; without it XML members are listed only

method : public : SetRowElement(row_element:String) ~ Nil

Parameters

NameTypeDescription
row_elementStringelement name