Space Data Module SDK

FlatSQL streaming standard

This document defines the recommended standard for getting streamed FlatBuffer payloads into FlatSQL-backed storage while keeping the same model portable across browser, server, OrbPro, and WasmEdge.

Short Version

Use three layers, not one:

  1. Transport stream: little-endian u32 size prefix + one FlatBuffer payload per frame
  2. Module invoke ABI: SDS $PIV request/response envelopes with TAB frame descriptors
  3. Durable storage identity: host-owned append-only row handles and host-owned runtime regions

Do not treat a module-owned mutable FlatSQL database as the canonical ABI. That can exist as an adapter or example, but it is not the durable cross-host contract.

Canonical Transport

The canonical transport for streamed FlatBuffers is:

Each payload must carry a readable FlatBuffer file identifier. The runtime uses that identifier to derive the durable schemaFileId.

This transport is intentionally separate from module invocation. It is the right shape for:

It is not the same thing as a SDS $PIV invoke envelope.

Canonical Durable Storage

The durable storage model is host-owned.

Standards records are addressed by:

Rules:

High-performance derived runtime state is addressed by:

Rules:

The stable storage ABI lives in schemas/HostStorageAbi.fbs.

Canonical Module ABI

When streamed payloads must cross into a module, use the invoke ABI:

Each input frame should preserve:

This is the correct way to tell a module "these frames belong to the same logical stream."

It is not a byte-streaming transport. The current invoke ABI is still buffer-oriented:

So a single 1 GiB SDS $PIV invoke request is not the intended path.

FlatSQL-Specific Rules

If the goal is "stream FlatBuffers into FlatSQL", standardize the host-side behavior first:

  1. split the outer size-prefixed transport stream into payload frames
  2. derive schemaFileId from the FlatBuffer file identifier
  3. append immutable host-owned rows
  4. maintain host-side logical indexes or latest-record projections as needed
  5. expose query and row-resolution APIs on top of that store

That means the durable primitive is append, not upsert.

upsert_records is allowed as a higher-level adapter surface, but only if it is defined as:

If a module or flow needs fast numeric state, keep that data in runtime regions, not in FlatSQL tables.

Browser and WasmEdge

The same transport/storage model should be used in both:

Use module invocation for:

Use host-side stream ingest for:

The new helper for this outer transport path is createFlatBufferStreamIngestor(...), exported from space-data-module-sdk/runtime-host.

Streaming Into A Module-Owned FlatSQL Engine

If the goal is not a host-owned durable store, but a stateful SDN module that imports flatsql internally and owns its own in-memory query state, use a persistent direct-surface module instance plus chunked binary invokes.

Rules:

The SDK helper for this path is createModuleFlatBufferStreamPump(...), exported from space-data-module-sdk/testing and the browser/root package surfaces.

That helper:

Example:

js
import {
  createBrowserModuleHarness,
  createModuleFlatBufferStreamPump,
} from "space-data-module-sdk";

const harness = await createBrowserModuleHarness({
  wasmSource,
  surface: "direct",
});

const pump = createModuleFlatBufferStreamPump({
  harness,
  methodId: "upsert_records",
  portId: "records",
  maxFramesPerInvoke: 64,
  typeResolver(_payload, context) {
    return {
      acceptsAnyFlatbuffer: true,
      fileIdentifier: context.rawFileIdentifier,
    };
  },
});

await pump.pushBytes(chunkA);
await pump.pushBytes(chunkB);
await pump.finish();

This is the right shape for OrbPro-style browser ingest when:

It is still batch-oriented per invoke, but it removes the architectural anti-pattern of building one monolithic request envelope for a long-running stream.

Example Import Path

js
import {
  createFlatBufferStreamIngestor,
  createRuntimeHost,
} from "space-data-module-sdk/runtime-host";

const host = createRuntimeHost();
const ingestor = createFlatBufferStreamIngestor({
  rows: host.rows,
});

ingestor.pushBytes(chunkA);
ingestor.pushBytes(chunkB);
ingestor.finish();

const rows = host.rows.listRows("OMM");

Performance Guidance

There are two distinct performance questions:

  1. Outer transport ingest throughput
  2. Module invoke throughput

Do not mix them into one benchmark.

Outer Transport Benchmark

Use the runtime-host stream ingestor.

Commands:

bash
npm run test:stream-ingest
npm run benchmark:stream-1gib

Optional tuning:

The 1 GiB benchmark is env-gated on purpose. It is a local stress path, not a default suite member.

Module Invoke Benchmark

Benchmark direct invoke separately with many smaller requests and a tiny response. Treat the current invoke ABI as batch-oriented, not as a raw stream transport.

Recommended total sizes:

Avoid a single 1 GiB request envelope on the current codepath.

For the resident module-ingest shape, use:

bash
npm run test:module-stream
npm run benchmark:module-stream-1gib

That benchmark exercises the chunked module stream-pump path, not a single huge invoke buffer.

Current Non-Canonical Example

examples/flatsql-store-local is still useful as an adapter example, but it is not the canonical durable identity model. It exposes a module-owned mutable logical database. The canonical model for cross-host durability is host-owned rows plus host-owned runtime regions.

Body-Reference Delivery ("deliver":"ref")

Loop C.5c adds an OPTIONAL near-zero-copy egress mode for capability hostcalls whose result is an aligned stream that the module passes through VERBATIM to an HTTP response body (the retrieval flow's flatbuffer branch). The election is always the GUEST's; hosts never invent it.

  1. The module adds "deliver":"ref" to the hostcall payload (storage.flatsql_query_stream / storage.flatsql_epoch_stream).
  2. A ref-capable host keeps the materialized bytes in ITS memory, registers them on the calling instance's hostcall-bridge body-ref registry, and answers with NO binary segment:
json
   {"ok":true,"result":{"rows":N,"columns":M,
     "ref":{"token":T,"size":S,"frames":F,"fnv1a64":"<16 hex>"}}}

A host that does not understand deliver simply returns the byte segment as always; modules MUST fall back to verbatim byte passthrough.

  1. The module forwards the reference in-band as a small JSON descriptor frame {"$sdnbodyref":1,"token":T,"size":S,"frames":F,"fnv1a64":"…"} (an aligned stream can never collide: it starts with a u32le size prefix, not {).
  2. foundation/http-respond recognizes the descriptor on its body port and emits $HTR with BODY_REF_TOKEN/BODY_REF_SIZE set and NO inline body (schema HttpResponseAbi.fbs).
  3. The host egress substitutes the byte buffer registered under the token (Go: flowrt.htrPipe + modulert.HostBridge.TakeBodyRef; JS: createBodyRefRegistry from space-data-module-sdk/http). Tokens are single-use; hosts drop unconsumed references at end-of-exchange (304 and error paths never consume theirs).

The stream bytes therefore never enter the flow's linear memory: the only byte movement left on a warm request is the host's socket write.

Source of truth for this page: docs/flatsql-streaming-standard.md. Regenerate with npm run build:docs.

SDN Stack

Connected sites

Standards Canonical schemas, record contracts, and generated bindings. FlatBuffers Binary encoding, schema tooling, and runtime documentation. FlatSQL SQL-style queries over FlatBuffer-backed datasets and streams. SDN Distributed publication, discovery, delivery, and marketplace infrastructure. Module SDK WASM module packaging, validation, and host compatibility tooling.