Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Open more actions menu

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
26 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

grange

A machin-native document database — agent-first, single binary, crash-safe by construction.

grange is a document store written in pure MFL that pairs with machin apps the way SQLite pairs with C: embed the engine (src/engine.src) directly in your binary, or drive the standalone CLI. No server, no dependencies, no cgo — one ~75 KB static binary.

  • Agent-first: JSON-only stdout, typed errors on stderr, semantic exit codes (80–119), guide + help-json introspection, per cli-specs. No human UI, ever.
  • Crash-safe: every commit is one immutable, checksummed WAL chunk. kill -9 at any moment leaves exactly the committed prefix — proven by make crash (5 rounds of mid-flight SIGKILL, recovered counts are exact commit-batch multiples).
  • Faster than SQLite on every indexed workload (100k docs, make bench, both engines indexed on the same field, same box):
metric grange SQLite
bulk insert, 2 indexes maintained 278k docs/s 25k rows/s
point get (avg of 1000) 5 µs 17 µs
indexed count × 1000 <1 ms (O(1) register) 1,937 ms
indexed find + fetch 33k docs 5 ms 111 ms
group-by count/sum/avg × 1000 <1 ms (O(1) registers) 49,114 ms
range count (score>=900) × 1000 <1 ms (after a one-time 79 ms sort) 257 ms (indexed)
full scan, no index (worst case) 61 ms 8 ms
cold open, 100k + index rebuild 309 ms

The one row SQLite wins is the unindexed scan (typed columns beat per-doc JSON extraction); the answer is grange index — one command, and that query class becomes O(1)/O(bucket) forever. Aggregate registers (declare --sums on an index) keep per-group count/sum/avg maintained incrementally at write time — a group-by answer costs a map lookup, which is why the agg row is not a typo.

Model

A database is a directory; a collection is a subdirectory. Docs are minified JSON keyed by id, held in memory, persisted as:

<db>/<coll>/seg-<gen>.grg       immutable compacted snapshot (generation gen)
<db>/<coll>/wal-<gen>-<n>.grg   immutable WAL chunk, one per commit

Every .grg file ends with a #|<nrecs>|<sha256:12> trailer. MFL has no file append or rename, so grange never mutates a file: a commit writes a fresh chunk, compaction writes a fresh segment (verified by re-read before anything is deleted). Recovery = load the newest valid segment, replay its valid chunks in order, drop anything torn.

Use

grange put  --db ./data --doc '{"name":"ada","status":"active"}'   # -> {"ok":true,"data":{"id":"..."}}
grange get  --db ./data --id <id>
grange find --db ./data --where status=active --limit 50
grange del  --db ./data --id <id>
grange count --db ./data --where status=active
grange index --db ./data --field status --sums score   # declare once: find/count O(bucket|1), agg O(1)
grange index --db ./data --field score --range         # sorted projection: > < >= <= in O(log n)
grange agg --db ./data --group-by status --sum score   # per-group count/sum/avg (--minmax f for min/max)
grange compact --db ./data        # fold WAL chunks into a fresh segment
grange stats --db ./data
grange guide                      # the machine-readable manual

--where clauses AND together and support = plus numeric > < >= <=: --where "status=active,score>=100".

Or run it as a server (grange serve --db ./data --port 4444) — same operations over HTTP/JSON with bearer-token auth (--token / GRANGE_TOKEN, else one is generated and printed at startup):

curl -X POST :4444/put -H "Authorization: Bearer $T" -d '{"doc":{"status":"active","score":9}}'
curl ":4444/find?where=score>=5" -H "Authorization: Bearer $T"    # /get /del /count /agg /index /stats /compact /health

The server is multi-collection (?coll= / body "coll", GET /collections lists them) and single-actor by construction: a sequential accept loop with zero goroutines, so there is nothing to race on — and machin's inferred data-race analysis (machin check, no annotations) verifies that on every build. Collection switching is O(1): MFL maps are reference types, so parking and restoring a collection's whole state is a handful of map assignments. Concurrent-reader serving is deliberately future work.

Hosted: grange.intrane.fr

A managed instance runs at https://grange.intrane.fr, with a read replica at https://read.grange.intrane.fr (same tokens, GET routes; writes 403 and belong on the primary). Signup is self-serve and agent-first — a peage wallet is the only credential:

curl -X POST https://grange.intrane.fr/tenants -H "X-Peage-Wallet: pw_..." -d '{"name":"my agent"}'
# -> {"tenant":"t...","token":"gt_...", "pricing":{...}}

Your gt_ token scopes every route to an isolated namespace with multiple databases (?db=, default default), each holding collections, indexes and its own WAL. Client SDKs for Python (pip i grange-db), Node.js (npm i grange-db), Go, and machin live in sdk/, with bulk writes (putMany — newline-delimited ops, one commit, all-or-nothing, ~263k docs/s measured over HTTP). Pricing: pay-as-you-go storage, €0.15/GB/month above 50 MB free, accrued continuously and charged to your wallet via peage (min charge 5 cents; GET /usage shows bytes, accrual, and charges at any time). No subscription, no card on file — fund the wallet once, the meter does the rest.

Embedded, from any machin app:

machin encode framework/flags.src src/engine.src yourapp.src > app.mfl && machin build app.mfl -o app
ok, err := gr_open("./data", "users")
gr_put("u1", "{\"name\":\"ada\"}")
gr_commit()                        // durable: one WAL chunk
doc, found := gr_get("u1")

Cold storage (disk-resident collections)

grange cold --db d --coll archive (or POST /cold) converts a collection to disk-resident: hash-partitioned page files (the same checksummed write-once format), a bounded memtable, and streaming scans. Measured at 200k docs: 4.4 MB RSS vs 89.9 MB hot in a fresh process, point gets read exactly one page file per run (3 ms cold-start included), scans and compaction stream one page group at a time. Trade-offs, enforced: no secondary indexes or TTL docs on cold collections (aggregate scans still work), and gets go from ~µs to ~100 µs. Hot stays the default — cold is for data bigger than your RAM budget.

Read replicas

The immutable WAL doubles as a replication log, so replicas need no new machinery — and no shared memory, so the race-freedom proof stands:

grange serve --db ./data --follow --port 4445   # local read-only follower: refreshes from the db dir before every read
grange follow --from https://grange.intrane.fr --rtoken gt_... --db ./replica   # LIVE local replica of a hosted db over the /watch feed

follow resyncs via /export?format=lines when behind, then applies watch deltas (puts + deletes), committing them to its own local WAL. make crash-grade durability applies to the replica too, because it IS a grange db.

Build & verify

make build    # needs machin >= 0.108
make verify   # check + tests (69) + 100k bench + crash harness

Scope & honesty (M5)

  • Whole dataset + indexes live in memory (memtable = the db); segments make cold open fast, not memory small. Steady-state RSS at 100k docs + 1 index is ~120 MB (fresh process); the bench process peaks at ~440 MB from MFL arena temporaries — a memory diet is the standing target.
  • --where supports equality + numeric ranges. Equality clauses use buckets; a single range clause on a --range field uses the sorted projection (built lazily on the first range query after a write — the build cost is the first query's, honestly). Multi-clause range queries scan. Aggregate registers cover count/sum/avg; --minmax computes min/max on the scan path.
  • Durability is process-crash-exact (proven by make crash), OS-crash best-effort (no fsync builtin in MFL yet).
  • grange serve handles one request at a time (correctness first) across any number of collections and tenants. Metering charges storage only (no per-query fees yet); billing sweeps piggyback on requests every 6h. Concurrent readers are next.

MIT.

About

machin-native document database — agent-first, crash-safe, faster than SQLite on indexed workloads

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Morty Proxy This is a proxified and sanitized view of the page, visit original site.