FlatSQL

Specification

FlatSQL store format 5

Each feed is one linear stream of size-prefixed FlatBuffers. Sibling metadata, checkpoint and segment frames make the stream the only authority: every index is a cache rebuilt from it, every record is a raw-leaf IPFS block pinned where it lies, and a backup is a CARv2 of the feed's root.

[u32 LE size][FlatBuffer] $FMB · $FCK · $FSG CIDv1 raw · sha2-256 CARv2 backups SDS 1.256.0

#Scope

This page specifies store format 5 of FlatSQL's partition store: a feed's bytes on disk, what an acknowledgment promises, how a feed is rebuilt from its own stream, and how a feed maps to IPLD (per-record CIDs, the feed root DAG and CARv2 backups). The node's side of IPFS (filestore registration, providing and pin policy) is summarized in §11.

#0Principles

  1. The stream is the only authority. Each feed is one linear stream of [u32 LE size][FlatBuffer] frames, appended in order. Everything needed to rebuild the feed is in those frames: the records, their metadata and the checkpointed state. Everything else is a cache derived from the stream: the SQLite index, the type index, the views, the IPLD link nodes and the CARs. Any of it can be deleted and rebuilt.
  2. Records are verbatim. A record frame holds exactly the bytes that arrived. No envelope, header or trailer is ever put around a record. Metadata goes in sibling frames.
  3. One record is one contiguous byte range. The raw-leaf CIDv1 (sha2-256) of that range is the record's IPFS block. Ranges are never changed while a filestore entry can name them.
  4. Every frame is a standard FlatBuffer. Each frame is identified by its 4-byte file identifier. The non-record frames are SDS tables (§3). A stock FlatBuffers reader walks the file from byte 0 and dispatches on the identifier. It needs no FlatSQL code.
  5. One format. A store in an earlier format is refused with P4_E_FORMAT, and a node rebuilds it by re-syncing its feeds.
  6. Each value lives in exactly one place. A value that can be re-derived from the record bytes is not stored again: the epoch and the object key are re-extracted on rebuild. $FMB holds only what the bytes cannot give. Between compactions, checkpoint segments are derived snapshots that can be discarded: the authoritative data is the records plus their $FMB frames. A compaction drops the $FMBs of dead rows. After that, the state those rows left behind (instance and token totals and times, dictionary ids) lives only in the new generation's segments, which are then authoritative for it.

#1Files

<data>/fsql4/
  STORE                    64 B, magic FSQ4, format 5 (any other format: P4_E_FORMAT)
  T/<TYPE>.spec            registered spec TLV
  T/<TYPE>.idx             type index: CACHE (feed and token registries, rebuilt from the streams)
  T/<TYPE>.fts             full text: CACHE
  P/<TYPE>/<feed>[.<gen>].fsdata   THE FEED: the linear frame stream (§2)
  P/<TYPE>/<feed>.db       SQLite index: CACHE (optional per workload, §7)

#2The stream

#2.1Frame bytes

frame at offset F:
  F+0  u32 LE  N            size of the FlatBuffer; no flag bits, N >= 8
  F+4  N bytes FlatBuffer   [u32 root uoffset][4-byte file identifier][...]
next frame at F + 4 + N; the file ends exactly at a frame end.
F+0u32 LE Nsize prefix
F+4u32 root uoffsetFlatBuffer starts
F+8"$FMB"file identifier
F+12 … F+4+Ntables, vectors, stringsthe rest of the FlatBuffer
One frame. Its offset is F; its range, the bytes a CID names, is [F+4, F+4+N). The next frame starts at F+4+N.
KindIdentifierWritten whenHolds
recordthe type's ($OMM, $WXF, ...) or a sealed envelopea new record or new bytes enter the feedthe record bytes exactly as they arrived
metadata$FMBonce per feed per commit round, after that round's record frames; also alone for removalsthe round's rows, dictionary additions and removals (§3.1)
checkpoint$FCKat stream creation (genesis), then on the cadence in §6the feed's identity, counters and live segment list (§3.2)
segment$FSGat checkpoints and in LSM mergesan immutable sorted state segment (§3.3)

#2.2Record frames

#2.3What a stock reader sees

#3Metadata, checkpoint and segment tables

The three non-record kinds are SDS standards in the FlatSQL family, beside FSB and FSO: FMB (batch metadata), FCK (checkpoint) and FSG (segment), as of SDS 1.256.0. The schemas below are those standards. No earlier SDS table fits these roles:

Conventions:

#3.1$FMB: batch metadata (one per feed per commit round)

FlatBuffers schema
// FMB: FlatSQL batch metadata. Sibling of the record frames it commits; never holds record bytes.
enum fmbDictKind : ubyte { TOKEN, NODE, BATCH, CONTENT_KEY, URL, OBJECT_KEY }
enum fmbRemoveReason : ubyte { DELETE, SUPERSEDE, CAT_SUPERSEDE, QUOTA, RETENTION, MOVED_OUT }

/// One row = one delivery of one record to this feed (rid = SEQ << 16 | ROW_J). 40 bytes.
/// No epoch or object key: both are re-extracted from the record bytes (principle 6).
struct FMBRow {
  OFFSET:ulong;          // F of the record frame (this round's, or an earlier shared frame)
  SEQ:ulong;             // the record's seq in the type's seq space
  SOURCE_TS:long;        // the copy's source timestamp: a PUT field (tag 53 entry), not in the bytes
  LENGTH:uint;           // N of the record frame
  GROUP:ushort;          // index into GROUPS
  ROW_J:ushort;          // j of the rid
  FLAGS:uint;            // bit0 NEW_FRAME (frame written this round), bit1 SEALED, bit2 LARGE,
                         // bit3 MOVED_IN (local -> feed, seq kept), bit4 COMPACTED (re-emitted by a compaction),
                         // bit5 COPY, bit6 RETAG, others 0
  RESERVED:uint;         // 0
}
/// What a call's rows share. 24 bytes.
struct FMBGroup {
  DELIVERED_AT:long;     // when this feed delivered the rows
  NODE_ID:uint;          // NODE (producer, peer)
  BATCH_ID:uint;         // BATCH (batch, producer peer, producer key)
  CONTENT_KEY_ID:uint;
  URL_ID:uint;
}
struct FMBRemove { SEQ:ulong; ROW_J:uint; REASON:fmbRemoveReason; }   // ROW_J 0xFFFFFFFF: every row of SEQ
struct FMBUrl    { SEQ:ulong; ROW_J:uint; URL_ID:uint; }               // a duplicate delivery moving a row's url
table FMBEntry   { KIND:fmbDictKind; ID:uint; TEXT:[string]; }         // TOKEN [token, peer]; NODE [producer, peer];
                                                                       // BATCH [batch, ppeer, pkey]; others [value]
table FMBSignature { ROW:uint; BYTES:[ubyte]; }                        // the row's signature
table FMBSealed  { ROW:uint; EPOCH:long = -9223372036854775808;        // ciphertext cannot be re-extracted, so a sealed
                   OBJECT_KEY_ID:uint; LEAF_DIGEST:[ubyte];            // row's epoch and key live here only; LEAF_DIGEST:
                   COLUMNS:[ubyte]; }                                  // ciphertext sha2-256; COLUMNS: its extracted values
table FMBIdentity { ROW:uint; HASH:[ubyte]; }                          // IQC ingest identity (32 B)
table FMBLarge   { ROW:uint; ROOT_CID:[ubyte]; ROOT_NODE:[ubyte]; LEAF_DIGESTS:[ubyte]; }  // §4.2

table FMB {
  SELF_OFFSET:ulong;
  PREVIOUS:ulong;            // F of the previous non-record frame
  FEED_ID:uint;
  COMMITTED_AT:long;         // the round's clock (restamps instances on removals)
  TYPE_SEQ_HIGH:ulong;       // the type's highest seq issued so far (seqs never go back)
  ROWS:[FMBRow];             // in plan order; at most 65,536 per frame (a round splits beyond)
  DIGESTS:[ubyte];           // 32 B per NEW_FRAME row, in row order: the record CID's digest
                             // (a shared-frame row takes the digest of the frame it names)
  GROUPS:[FMBGroup];
  DICTIONARY:[FMBEntry];     // entries first used by this round (deltas)
  REMOVALS:[FMBRemove];
  URLS:[FMBUrl];
  SIGNATURES:[FMBSignature];
  SEALED:[FMBSealed];
  IDENTITIES:[FMBIdentity];
  LARGE:[FMBLarge];
}
root_type FMB;
file_identifier "$FMB";

What the vectors hold:

The instances, the token counts and the counters are folds over the rows.

Every change is an appended frame; nothing is changed in place:

#3.2$FCK: checkpoint (small; the state lives in segments)

FlatBuffers schema
enum fckSegmentKind : ubyte {          // the FSG kinds, §3.3
  DICTIONARY, LATEST, INSTANCES, TOKENS, TOMBSTONES,
  ROWS_BY_SEQ, ROWS_BY_OBJECT_EPOCH, ROWS_BY_CID, ROWS_BY_W, ROWS_BY_AT,   // the row orders, §7
  COLUMN = 64, ZONE_MAP = 65, ROLLUP = 66                                 // reserved: analytics, 64..127
}
struct FCKCounters {                   // the feed's counters
  ROWS:ulong; RECORDS:ulong; BYTES:ulong; RECORDS_NO_EPOCH:ulong;
  COPIES:ulong; COPY_BYTES:ulong; LIVE_FRAME_BYTES:ulong;
  MIN_SEQ:ulong; MAX_SEQ:ulong; MIN_EPOCH:long; MAX_EPOCH:long;
  MIN_TS:long; MAX_TS:long; MIN_AT:long; MAX_AT:long;
}
struct FCKSegment {                    // one live FSG frame. 48 bytes
  OFFSET:ulong; LENGTH:uint; KIND:fckSegmentKind; LEVEL:ubyte; RESERVED:ushort;
  MIN_SEQ:ulong; MAX_SEQ:ulong; COVERS_THROUGH:ulong; ENTRIES:ulong;
}
table FCK {
  SELF_OFFSET:ulong;
  PREVIOUS:ulong;
  FEED_ID:uint;
  TYPE:string; PROVIDER:string; SOURCE:string;   // the feed's identity: the type index is rebuilt from these
  GENERATION:uint;
  COMMITTED_AT:long;
  APPLIED_THROUGH:ulong;      // end (F+4+N) of the last FMB folded into the segments
  TYPE_SEQ_HIGH:ulong;
  COUNTERS:FCKCounters;
  SEGMENTS:[FCKSegment];
  SEGMENT_DIGESTS:[ubyte];    // 32 B per segment: its raw-leaf CID digest
  COMMITS_ROOT_CID:[ubyte];   // the commits tree (§4.3) as of APPLIED_THROUGH
  PREVIOUS_FEED_ROOT_CID:[ubyte];
  RECORD_BYTES:ulong;         // each record's smallest copy, summed; 0 when COUNTERS.ROWS is 0
  MIN_W:long; MAX_W:long;     // w bounds over the rows; 0 when COUNTERS.ROWS is 0 (read as none)
}
root_type FCK;
file_identifier "$FCK";

#3.3$FSG: immutable state segment

FlatBuffers schema
struct FSGPair {             // the one sorted entry shape. 32 bytes
  KEY_HI:ulong; KEY_LO:ulong;
  RID:ulong;                 // SEQ << 16 | ROW_J
  ROW_REF:ulong;             // (F of the FMB holding the row) << 16 | row index in it
}
struct FSGLatest { OBJECT_KEY_ID:uint; RESERVED:uint; EPOCH:long; RID:ulong; ROW_REF:ulong; }  // 32 B
struct FSGInstance {         // one instance row. 88 B
  BATCH_ID:uint; CONTENT_KEY_ID:uint; RECORDS:ulong; BYTES:ulong; MIN_W:long; MAX_W:long;
  MIN_SEQ:ulong; MAX_SEQ:ulong; FIRST:long; UPDATED:long; MAX_AT:long; MAX_TS:long;
}
table FSG {
  SELF_OFFSET:ulong;
  PREVIOUS:ulong;
  FEED_ID:uint;
  KIND:fckSegmentKind;       // which vector is filled
  LEVEL:ubyte;
  MIN_SEQ:ulong; MAX_SEQ:ulong;
  COVERS_THROUGH:ulong;
  REPLACES:[ulong];          // F of the segments an LSM merge replaced
  ENTRIES:[FMBEntry];        // DICTIONARY, sorted by (KIND, ID)
  LATEST:[FSGLatest];        // LATEST, sorted by OBJECT_KEY_ID
  INSTANCES:[FSGInstance];   // INSTANCES, sorted by (BATCH_ID, CONTENT_KEY_ID)
  TOKENS:[FMBEntry];         // TOKENS: per-producer counts as TEXT [token, n, bytes, mints, maxts, maxseq]
  PAIRS:[FSGPair];           // TOMBSTONES (KEY_HI = rid) and ROWS_BY_*, sorted by (KEY_HI, KEY_LO, RID)
  COLUMN:[ubyte];            // reserved (analytics)
  INSTANCE_URL_IDS:[uint];   // parallel to INSTANCES, each instance's latest url (a URL dictionary id; 0 = none)
}
root_type FSG;
file_identifier "$FSG";

How the row orders fill PAIRS. Here w is the record's epoch, or its source timestamp when it has no epoch; at is the delivery time; cp is the first 8 bytes of the CID key.

KindKEY_HIKEY_LO
ROWS_BY_SEQrid0
ROWS_BY_OBJECT_EPOCHobject idepoch XOR 2^63
ROWS_BY_CIDthe 8-byte key prefix (cp)the next 8 bytes
ROWS_BY_Ww XOR 2^630
ROWS_BY_ATat XOR 2^630

#4IPFS mapping

#4.1Per-record blocks

#4.2Records over 1 MiB

#4.3The feed root DAG

All link nodes are dag-pb, CIDv1, sha2-256. FlatSQL computes them as a pure function of the frames: they are never FlatSQL disk state. The node keeps them in its blockstore when exposure is on, and the CAR export emits them.

feed root                       dag-pb UnixFS Directory, links sorted by name
├── "checkpoint" -> the newest $FCK frame              raw leaf, in situ
├── "commits"    -> commits tree                       dag-pb, no Data; fanout <= 1,024
│     └── ... -> commit unit (one per FMB, stream order)  dag-pb, no Data, links in stream order:
│                  NEW record frames (raw leaf | §4.2 root), then the $FMB frame (raw leaf)
├── "segments"   -> node linking each live $FSG frame (raw leaf, in situ), in FCK.SEGMENTS order
└── "columns"    -> reserved for analytics column segments (absent)

#5Acknowledgment and durability

Rule. For one call, the writer:

  1. appends its record frames and then its feed's $FMB to each touched stream;
  2. syncs each touched stream once (in parallel when several);
  3. commits the SQLite index without a device sync (WAL, synchronous=NORMAL), for read-after-write;
  4. publishes and acknowledges.

What an acknowledgment promises. The record bytes and every column of their rows are durable in the stream. The index may be behind after a power loss, and recovery (§9) restores it from the $FMB frames.

Sync cost. A call pays one sync per touched stream. Syncing the index as well would make it two, so a single-feed call drops from 2 syncs to 1. A call that also creates a feed pays once for the feed: the new file, its genesis $FCK and a directory sync. A new token adds no sync, because it lives in the $FMB dictionary.

SQLite never syncs on the acknowledgment path. The index files open through a VFS shim (flatsql_p4_cache) whose xSync is a no-op outside WAL checkpoints and connection close, so a WAL restart's header sync is skipped too. A power loss can therefore drop index commits, but never acknowledged data. Checkpoints keep SQLite's own sync order.

Split rounds. A round whose feed rows exceed 65,536 writes several $FMBs. The acknowledgment follows the sync of all of them. Recovery may apply a durable prefix of an unacknowledged round; such records were never acknowledged and are kept, as any durable but unacknowledged write is.

Order of persistence. The index commit is written only after its stream sync returned. A persisted index commit therefore never names bytes the stream lost. The index's mark is the end of the last $FMB it applied.

Background work keeps its own syncs off the acknowledgment path:

Measured: a checkpoint preparing does not slow the acknowledgment. PUT acknowledgment latency, p50/p99:

p50p99
No checkpoint preparing8.80 ms13.95 ms
A checkpoint preparing9.04 ms10.44 ms

Installing the prepared checkpoint takes about 73 ms, once, on the round where the appender installs it.

Basis: FlatSQL 3.10.0's replay gate (REPLAY_CHECKPOINT_PREPARE, native):

Registration follows the acknowledgment:

#6Checkpoints and segments

Cadence. A checkpoint is written when the first of these holds:

Writing one.

  1. Fold the rows since APPLIED_THROUGH into new level-0 segments, one per changed kind: dictionary entries, latest-per-object, instances, tokens, tombstones, and the row orders where enabled (§7).
  2. Append the segments, then the $FCK that lists every live segment.

Off the PUT path.

LSM.

Open. Open reads the newest $FCK, its segments and the tail $FMBs after APPLIED_THROUGH. No SQLite file is needed for the counters, dictionary, instances, tokens or latest-per-object.

#7Index choice

#8Backups and archives: CARv2

A backup of a feed is the CARv2 of its feed root. A store or type backup is the CAR of a dag-pb UnixFS Directory <TYPE>/<feed>, whose links are the feed roots.

Bytes.

The same state always gives the same CAR bytes. There is no custom container.

Restore.

Backups are tables themselves.

#9Crash recovery from the stream tail

The commit point. A commit point is a verified non-record frame on the commit chain. An $FMB verifies when all of these hold:

An $FCK or $FSG verifies on the first three. The digest check makes recovery safe even when a crash persisted later pages before earlier ones.

Open, per feed:

  1. Start.
    • With an index: start at its mark. A stream shorter than its mark lost acknowledged frames, and the feed is quarantined (P4_E_CORRUPT, named).
    • Without one: find the newest $FCK by the tail search below and load its segments.
  2. Walk forward from the start, collecting record frames until a candidate commit point. If the candidate verifies, apply it with its seqs and metadata. Rows get no fresh seqs, and provenance is never blank.
  3. Cut the tail. At EOF, or at a torn or unverifiable frame, cut the stream back to the end of the last commit point. Those bytes were never acknowledged and never registered.
  4. Finish moves. A MOVED_IN row in a feed whose MOVED_OUT is missing in the local feed gets its removal $FMB appended to the local feed.
  5. Rebuild caches as needed:
    • the views from the index's staged rows;
    • a missing or damaged .db from the $FMB chain;
    • a missing .idx from the genesis frames.

Tail search (no index).

Rebuilding the SQLite index without its file.

#10Retention and compaction (pins and filestore first)

Retention, QUOTA_GC, DELETE, SUPERSEDE:

  1. Append the removal $FMB and sync. The decision is now durable.
  2. Remove the index rows. Reads stop answering them.
  3. Publish the drop list (CID, unit) and the next feed root at the next checkpoint (or at once when the feed is exposed).
  4. The node does pin update old-root -> new-root, removes the dropped CIDs' filestore entries (reference-counted across feeds, §11) and calls p4 release(feed, through-commit).
  5. Only released bytes may physically go (step 2 of a compaction). A feed that is not exposed is released implicitly.

The stream is append-only, so no byte under a live filestore entry is ever changed. Retention by arrival drops whole FSG segments (§6).

Compaction (a stream more than half dead, with at least 1 MiB dead):

  1. Write generation g+1: the genesis $FCK, then the live record frames in stream order in chunks of at most 65,536 rows, each chunk followed by its consolidated $FMB (rows COMPACTED, new offsets), then fresh segments and the final $FCK. Then sync.
  2. Repoint the index (one transaction) and publish the moved ranges and the new root.
  3. The node re-registers the same CIDs at their new (path, offset) and pin-updates to the new root. Record CIDs do not change, because records are verbatim. Then it releases generation g: release(feed, g) releases all of the immutable generation g whatever through-commit it names, and release(feed, through) on g+1 releases g once through reaches the compaction's final $FCK.
  4. Unlink g after the release and after the 30 s reader window.

A node that never releases blocks compaction, and the store reports it as a stat. It never loses data.

#11The node's side

The node, not FlatSQL, runs the IPFS side of a feed:

#12Costs and measurements

M = measured on a Mac Studio, native. E = estimated. The gate fixture holds two feeds: an OMM feed of 623,462 rows (a 204.7 MB stream of 328 B records) and a WXF sea feed of 1,720,179 rows (15.0 GB of 8.7 KB records).

PropertyHowCost
The disk holds a linear [u32 size][FlatBuffer] stream, appended in order, that any FlatBuffers reader opens§2: one stream per feed generation; four frame kinds told apart by identifier; no header, padding or trailermetadata frames are copied to an aligned buffer when read (open and rebuild only): negligible
A table rebuilds from FlatBuffers alone: sibling per-record metadata frames, never an envelope, plus periodic checkpoints§3.1 $FMB per feed per round (every row column, the dictionaries, removals); §3.2–3.3 $FCK and $FSG; SQLite, the type index and the views are caches$FMB = 72 B/row (40 B struct + 32 B digest) + 24 B/call + dictionary deltas. Gate fixture (M sizes): OMM 623,462 rows → 44.9 MB = +22% on a 204.7 MB stream (328 B records); WXF sea 1,720,179 rows → 123.9 MB = +0.8% on 15.0 GB (8.7 KB records), plus about 11 MB (E) for its 191,131 batch strings. Checkpoint segments: at most 25% of appended bytes by the cadence rule, typically about 1–2% (E); LSM write amplification about log4(segments) (E)
Records are pinned in situ: each record one contiguous range, raw-leaf CIDv1 (sha2-256) = record CID, Kubo filestore nocopy§2.2, §4.1: the stored CID is already the range's sha2-256 (kVerifyCid, on by default); the node registers (path, off, len, CID)write path: 0 extra hashing (the digest exists); 72 B/row in $FMB (above). Kubo side (E): about 150–200 B per filestore entry in its datastore, about 50–60% of OMM's stream bytes and about 2% of WXF's, so exposure is opt-in per feed
Records over 1 MiB are a DAG of raw-leaf sub-ranges§4.2: 1 MiB leaves; UnixFS root byte-equal to ipfs add --raw-leaves --chunker=size-1048576about 410 B root node + 32 B per leaf in FMBLarge; no fixture record exceeds 1 MiB (M: max 8,720 B)
Backups and archives are IPLD CARv2 (record blocks, index segments, checkpoint root DAG), and a backup is itself a table§8: deterministic CARv2 of the feed root; restore uses the same apply path as recovery; a restored CAR is a feedCAR overhead (E): about 40 B per block section + 40 B per index entry: OMM about 1.25x its live stream, WXF about 1.01x. dag-pb links about 44 B per record (in the node's blockstore, not FlatSQL's)
Roots are announced, not records; IPFS is per feed and off by default; restricted feeds are never registered or served§4.3, §11: FlatSQL computes roots and range tuples for every feed (cheap, and needed for backups); registration, providing and serving are the node's per-feed policyDHT load: one provider record per feed root per checkpoint, instead of one per record
FlatBuffer index segments may replace SQLite per workload, within about 10%§3.3, §6, §7: FSGPair sorted vectors, LSM, whole-segment retention, behind the same reader interface32 B per row per order (5 orders = 160 B/row), against SQLite's 193 B/row for OMM (M: 120.7 MB / 623,462) and 180 B/row for WXF (M: 309.8 MB / 1,720,179)
Room for analytics column segmentsfckSegmentKind 64–127 and FSG.COLUMN reserved; root link "columns" reservednone
One formatearlier formats are refused (P4_E_FORMAT); nodes rebuild by re-syncone re-sync per node
Acknowledgment and durability§5: one stream sync per touched stream; index commit unsynced (WAL NORMAL)M: 4.79 ms per sync (F_FULLFSYNC; gate: 16 syncs = 76.66 ms). Gate wxf_lead 746 ms with 128 syncs (613 ms) → 64 syncs, about −307 ms (−41%); gp_ingest 476 ms with 16 → 8, about −38 ms (−8%) (E from M)
Crash recovery from the tail§9: commit chain + digest-checked $FMBs; tail search within about 64 MiBopen without the index (E): newest $FCK + segments + tail of at most 64 MiB, about 1 s. Rebuilding the 15 GB fixture's SQLite index from the stream (E): one sequential read of 15.2 GB, about 1 µs/row of re-extraction and a bulk insert of 2.34M rows, about 15–30 s
Reads through IPFS or directnot a format change(E) direct warm pread 5–20 µs/record; through the local filestore +30–100 µs/block (datastore lookup, pread, Kubo's re-hash); remote = network round trip + that
The node pulls per acknowledged round, so a pull costs what changed, not the feed§4.3: RANGES from a commit walks only the nodes rewritten since; FEED_ROOT computes only the root; an earlier commits root comes from the forward cacheM (the gate's WXF sea feed, 110,915 blocks, against a 92-block feed of the same lane shape; medians of 7 rounds): FEED_ROOT 0.109 / 0.064 ms; RANGES from the prior commit 0.089 / 0.038 ms (14 blocks); earlier commits root 4 / 3 µs. The whole-closure RANGES on the sea feed costs 68.6 ms

#Measured in the implementation

Measured as each part of the format was built (FlatSQL 3.8.0 to 3.10.1): the replay gate's native medians against store format 4 (FlatSQL 3.7.2) as the baseline, and the 15 GB replay fixture.

PropertyMeasured
One stream sync per acknowledgmentwxf_lead 799 → 398 ms; gp_ingest 531 → 478 ms
$FMB size76 B per row
Crash recoverykill -9 in the middle of a lead (3 runs) and rebuilds with every .db deleted answer exactly
Open and rebuild with every index file deleted (15 GB fixture)checkpoint open 2.6 s; row-index rebuild 18.5 s; every answer equal
Checkpoint segments4.8% of an OMM stream (328 B records) and 0.7% of a WXF stream; checkpoints under 0.01%
Replay with checkpoints and segmentswxf_lead 826 → 431 ms; explorer warm 87.8 → 9.4 ms; sync_window warm 142 → 78 ms; latest warm 112 → 87 ms
Feed root DAG resident state88.9 B per row
Records over 1 MiBroot CIDs byte-equal to Kubo's ipfs add --cid-version=1 --raw-leaves --chunker=size-1048576
CARv2 exportipfs dag import takes the CAR byte for byte
A compacted feed with its index deletedrebuilds exactly, including urls that only dead rows wrote
Acknowledgment while a checkpoint preparesp50/p99 9.04/10.44 ms, against 8.80/13.95 ms with none; the install step about 73 ms on one round (§5)
Pulling one round from a 110,915-block feedRANGES 0.089 ms and FEED_ROOT 0.109 ms; the whole-closure walk 68.6 ms (§12)

Source: cpp/src/p4/FORMAT.md in the FlatSQL repository, commit d57ca6c92e. The schemas are the SDS standards FMB, FCK and FSG (SDS 1.256.0).