BJData

BJData is little-endian and self-delimiting, which is what makes reading it in place possible.

Size or speed

A document is either to be as small as it can be, or as quick to write and read as it can be. That is the one thing to choose, and everything the writer decides follows from it:

bjdata::encode(value);                        // prefer::size, the default
bjdata::encode<bjdata::prefer::speed>(value);

	prefer::size	prefer::speed
a number on its own	the narrowest marker that holds it exactly: 200 is U, 1.0 is a three-byte h	the marker of its own type: an int is l, a double is D
a range of numbers	a typed array, [$T#n, at the numbers' own type	the same
any other range	unbounded, [ … ]: a count costs bytes	counted, [#n: the reader sizes its container once

A range of numbers is a typed array whichever is preferred, at the width the numbers already have. Narrowing is for a value on its own, or in an array that is mixed anyway: narrowing a run of numbers would mean measuring it and then storing each element, where this is one copy to write, one to read, and a span to view. A std::vector<std::int32_t> of small values you want small is a std::vector<std::uint8_t>.

Either is read by the same reader, which takes whatever marker it finds. It is a template argument, so a build carries the code of the one it uses.

Zero copy, concretely

// [$u#U3 - three little-endian uint16s
if (auto samples = document["adc"].as<nonstd::unaligned_little_span<const std::uint16_t>>()) {
    auto peak = std::ranges::max(*samples);                              // no copy
    auto owned = std::ranges::to<std::vector<std::uint16_t>>(*samples);  // opt in to one
}

as<nonstd::unaligned_little_span<const T>>() requires the element marker to be exactly what T packs as, because zero copy demands an exact layout match. Decoding into a std::vector<T> of that type is one copy.

N-dimensional arrays

A dimension-array count becomes a strided view, so peeling a dimension is arithmetic, and the column-major form is a different set of strides over the same bytes.

#include <serpent/bjdata/ndarray.hpp>

if (auto grid = bjdata::as_ndarray(document["image"])) {
    grid->shape();                              // {480, 640}
    grid->at(12).at(34).value().as<int>();
    grid->at(12).flat<std::uint8_t>();          // a whole row, when contiguous
}

Splicing

view::payload_bytes() gives a value's own bytes, so a subdocument moves with no re-encoding:

bjdata::write_value(writer, document["calibration"]);   // marker, then one memcpy

Block notation

#include <serpent/bjdata/notation.hpp>

bjdata::block_notation(bytes);   // [S][U][5][hello]

A trace of the bytes, not the decoded value, so it catches grammar drift that comparing values would miss.

Scope

Draft 3 in full: every marker, all container forms, dimension-array counts, and the noop rules. E is rejected rather than skipped.

Strict grammar

A strong type must be fixed width, so $S, $H, $Z, $T and $F are rejected — which is what keeps every typed container O(1) to index and skip. Some libraries write arrays of strings as [$S#…, so reading its output needs a leniency mode this does not yet have.

Structure-of-Arrays tables

A draft 4 table — [${ schema }#count and the records back to back, or {${ with every record's first field before any record's second — is read every way a document is: checked by validate, which reads every record's booleans and dictionary and offset indices as a reader handle would, so that a table it passes is one every handle can read; into a std::vector<T> of a described type record by record at fixed offsets, into a value as an array of objects, and through reader handles, where a table is an array whose elements are objects with the schema's fields. Every schema field type is understood: fixed-width scalars, the T boolean byte, Z, padded S/H text, dictionaries, offset-table text, nested records and fixed runs. A member of the type that the schema has no field for keeps its value unless the type insists on it; a field the type has no member for is skipped. A table with more than one dimension is read flat, in record order. A record must take a byte at least: a schema of null fields and text of no length, whose records would take none, is refused as a strong type with no payload is, since a count of such records is held to no bytes.

Writing goes the other way from the type: a range of two or more records of a described type whose every member a schema can say — a number, a boolean, a character, text, a mapped enumeration, or a nested record or fixed run of those — is written as a table. The schema comes from the type, so it is the same for every document the type produces: numbers at their declared width, a mapped enumeration as a dictionary of its names (or the narrowest integer that holds its numbers), a plain enumeration as its underlying number. Text is the one thing decided from the data, per column: single characters as C, up to 255 distinct values as a dictionary, more as an offset table. Each record is then one piece, its members back to back with no keys and no markers, which is why a table of ten thousand numeric records writes in a third of the time of the array of objects and in a fraction of the bytes.

A type with an optional, a variant, a growable container or a value member is not a record; neither is a single element or an empty range. Those write as arrays of objects, as does anything when the writer is told { .tables = false } — for a consumer that reads draft 3 only, such as the DOM library. A table of more than one dimension is not written.