In-memory query engine over typed columns. Tables are filled row by row or loaded from sources (CSV files, folders, the program's own objects) and queried with SQL-like statements; aggregates, Describe() and Finder (file search) are included. Compile with -lib query.
ArchiveSource
A zip or tar archive as tables, without extracting it: a listing of its members as a tree (like FilesystemSource's), and each member in a format the library reads as a table of its own, named by the member's path ('data/runs.csv'; in SQL, "data/runs.csv" in double quotes). Members are read by extension: '.csv', '.tsv', '.jsonl' and '.ndjson', '.json', '.otab', and '.xml' once SetRowElement names the row element, each also with '.gz'. Other members, '.txt' included, are listed only; ReadBytes and ReadText read any of them.
The archive is a '.zip' (stored and deflate members, ZIP64, a self-extracting prefix), a '.tar' (POSIX ustar, pax and GNU long names) or a gzipped tar ('.tar.gz', '.tgz'), told apart by content rather than name, from a file, an https:// URL or bytes in memory.
Untrusted archives are safe to read. Nothing is written to disk. A member whose path is absolute or climbs out with '..' is refused, as are duplicate names in a zip and members whose data overlap. Each zip member is inflated to at most the size its header states and checked against its CRC32, so a zip bomb or a damaged member is an error naming it, never wrong data. Reading stops at 256 MB for one member and 1 GB for the whole archive unless SetLimits says otherwise. Encrypted members, compression other than deflate, and archives split across disks are reported as unsupported. Links are listed but never followed.
The listing table is named after the archive ('results' for 'results.zip'), with columns id, parent_id (empty at the top), depth (0 at the top), name, path, ext, is_dir, type ('file', 'dir', 'symlink', 'hardlink' or 'other'), size, compressed_size and method (zip only; method is 'stored', 'deflate' or the name of another), crc32 (zip only), modified and link (a link's target). Folders that the archive implies but does not list get rows too, so every member has a parent. A tar member repeated later in the archive replaces the earlier one's table, as extracting would.
Example
catalog := Catalog->New();
if(catalog->Load(ArchiveSource->New("results.zip"))) {
rs := catalog->Query("select path, size from results where ext = 'csv' order by size desc");
runs := catalog->Query("select name, avg(score) from \"2026/runs.csv\" group by name");
};Operations
- New
- ColumnNames
- FromBytes
- GetError
- GetMembers
- GetTables
- List
- Load
- ReadBytes
- ReadTable
- ReadText
- SetLimits
- SetName
- SetPattern
- SetRowElement
ColumnNames # function
Gets the listing's column names
function : ColumnNames() ~ String[]Return
| Type | Description |
|---|---|
| String | column names |
FromBytes # function
Makes a source over an archive already in memory
function : FromBytes(name:String, bytes:Byte[]) ~ ArchiveSourceParameters
| Name | Type | Description |
|---|---|---|
| name | String | the listing table's name |
| bytes | Byte | archive bytes |
Return
| Type | Description |
|---|---|
| ArchiveSource | source |
GetError #
Gets the reason the last read failed
method : public : GetError() ~ StringReturn
| Type | Description |
|---|---|
| String | error, Nil if none |
GetMembers #
Gets the paths of the archive's files, in archive order
method : public : GetMembers() ~ String[]Return
| Type | Description |
|---|---|
| String | member paths, Nil if the archive cannot be read (see GetError) |
GetTables #
Gets the tables the archive provides, the listing first; the members' columns are known only after reading them
method : public : GetTables() ~ TableSpec[]Return
| Type | Description |
|---|---|
| TableSpec | table shapes |
List #
Lists the archive's members as a table
method : public : List() ~ TableReturn
| Type | Description |
|---|---|
| Table | listing, Nil if the archive cannot be read (see GetError) |
Load #
Loads the listing and every member readable as a table into a catalog. Nothing is added unless every member reads. The scan hint is not pushed into members.
method : public : Load(catalog:Catalog, hint:ScanHint) ~ BoolParameters
| Name | Type | Description |
|---|---|---|
| catalog | Catalog | catalog |
| hint | ScanHint | ignored; the engine applies the query afterwards |
Return
| Type | Description |
|---|---|
| Bool | true if loaded |
New # constructor
Constructor; the listing table is named after the archive
New(path:String)Parameters
| Name | Type | Description |
|---|---|---|
| path | String | archive file or URL ('.zip', '.tar', '.tar.gz' or '.tgz') |
ReadBytes #
Reads one member's bytes, checked against its CRC32 and limits
method : public : ReadBytes(member:String) ~ Byte[]Parameters
| Name | Type | Description |
|---|---|---|
| member | String | member path, as the listing gives it |
Return
| Type | Description |
|---|---|
| Byte | bytes, Nil if missing or unreadable (see GetError) |
ReadTable #
Reads one member as a table, by its extension, whether or not it matches the pattern
method : public : ReadTable(member:String) ~ TableParameters
| Name | Type | Description |
|---|---|---|
| member | String | member path, as the listing gives it |
Return
| Type | Description |
|---|---|
| Table | the member's table (a JSON or XML member's child tables are left out), Nil if it cannot be read as one (see GetError) |
ReadText #
Reads one member as UTF-8 text
method : public : ReadText(member:String) ~ StringParameters
| Name | Type | Description |
|---|---|---|
| member | String | member path, as the listing gives it |
Return
| Type | Description |
|---|---|
| String | text, Nil if missing or unreadable (see GetError) |
SetLimits #
Sets how much may be decompressed: a member larger than 'member_bytes' is refused, and so is a read that would take the archive's total past 'total_bytes'. The defaults are 256 MB and 1 GB.
method : public : SetLimits(member_bytes:Int, total_bytes:Int) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| member_bytes | Int | most bytes one member may hold uncompressed |
| total_bytes | Int | most bytes all members read together may hold uncompressed |
SetName #
Sets the listing table's name
method : public : SetName(name:String) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| name | String | table name |
SetPattern #
Sets which members become tables: a pattern over member paths, where '*' matches any run of characters (slashes included) and '?' any one. All readable members by default.
method : public : SetPattern(pattern:String) ~ NilParameters
| Name | Type | Description |
|---|---|---|
| pattern | String | pattern, such as 'runs/*.csv' |