Digital Arsenal
FlatSQL

SQL queries over raw FlatBuffer storage. Zero conversion, real-time indexing, pure streaming.

Apache 2.0 licensed, free and open source.

Built on DA-FlatBuffers

CI Stars Forks

tj@digitalarsenal.io

Try Live Demo View Source
Scroll

Built for Streaming

No parsing. No conversion. Binary data arrives ready to query.

Raw Streaming Ingest

Pipe size-prefixed FlatBuffers directly to the engine. File identifiers route to tables automatically.

File ID Routing

Each FlatBuffer's 4-byte identifier routes it to the correct table. No external metadata needed.

Inline Index Building

B-tree indexes built during streaming ingest. No separate indexing phase. Query immediately.

Zero-Copy Access

Direct pointer access to FlatBuffer data. No deserialization. Maximum performance.

Overview

A database built for the real-time, decentralized web.

Why FlatSQL?

Modern apps receive data from everywhere

Sensors
P2P Networks
Edge Devices
Cloud APIs
Streams

Traditional Database

1 Receive binary data
2 Parse to objects
3 Convert to SQL types
4 Insert rows
5 Query → serialize back
VS

FlatSQL

1 Receive binary data
2 Store as-is
3 Query directly

FlatSQL keeps data in its native FlatBuffer format, routes it by source, and queries it with SQL. Ideal for isomorphic applications where the same code runs in browsers, workers, and servers—all sharing the same binary protocol.

Distributed IoT
Collaborative Editing
Real-time Analytics
Edge Computing

Data Flow Architecture

FlatBuffers
Size-prefixed binary data streams in via stdin, socket, or API
→
File ID Router
4-byte identifier routes each buffer to the correct table
→
SQLite Index
SQLite B-tree indexes for fast lookups
→
SQLite VTable
SQL interface with zero-copy field extraction
→
Query Results
Direct pointer access to FlatBuffer fields

Design Decisions

  • Zero Deserialization — Data stays in FlatBuffer format, extracted on-demand
  • Source Routing — Same schema, multiple sources; query by _source field
  • Append-Only — Optimized for streaming; no write amplification
  • SQLite VTable — Full SQL without data copying

Benchmarks vs SQLite

Ingest 2.5x faster
Point Query 1.8x faster
Zero-Copy 8x faster
FlatSQL
SQLite

Trade-offs

  • Zero-Copy API — 1.7µs lookups (1.4x faster than SQLite)
  • VTable SQL — 12.9µs queries (5-7x slower due to API limits)
  • Append-Only — No updates; use delete + re-insert
  • In-Memory — Disk persistence via export/import

See full trade-offs analysis →

Interactive Demo

Stream FlatBuffers and query with SQL in real-time.

FlatSQL Engine

Loading WASM...

Streaming Ingest

Records 0
Bytes 0
Rate/sec -
Stream data to see activity

SQL Query

Try:

JSON Schema IDL with x-flatbuffers Extensions

Generated by DA-Flatbuffers — used to initialize FlatSQL 
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "x-flatbuffers": {
    "namespace": "demo",
    "root_type": "demo_User",
    "file_ident": "USER"
  },
  "definitions": {
    "demo_User": {
      "type": "object",
      "x-flatbuffers": { "type": "table" },
      "properties": {
        "id": {
          "type": "integer",
          "x-flatbuffers": { "id": 0, "base_type": "int32" }
        },
        "name": {
          "type": "string",
          "x-flatbuffers": { "id": 1 }
        },
        "email": {
          "type": "string",
          "x-flatbuffers": { "id": 2, "key": true }
        },
        "age": {
          "type": "integer",
          "x-flatbuffers": { "id": 3, "base_type": "int32" }
        }
      }
    }
  }
}

Simple API

Stream format: [4-byte size LE][FlatBuffer with file_id]...

// Create database from schema
const db = new FlatSQLDatabase(schema, "mydb");

// Register file identifiers for routing
db.registerFileId("USER", "User");
db.registerFileId("POST", "Post");

// Register data sources
db.registerSource("satellite-1");
db.registerSource("ground-station");

// Stream size-prefixed FlatBuffers to a source
const bytesConsumed = db.ingest(streamData, "satellite-1");

// Query with SQL (filter by _source)
const result = db.query("SELECT * FROM User WHERE _source = 'satellite-1'");