π¦ Lightweight binary schema encoding β define a schema, encode to bytes, decode back. Zero dependencies.
JSON is great β until you're sending thousands of small messages over WebSockets, WebRTC data channels, or storing millions of tiny records. Then every message pays for its own key names, quotes, and decimal digits:
// JSON β 91 bytes:
{"type":64,"sub":8,"subjectPeerId":918273645,"name":"peer-x1","vals":[1,22,333,4444,55555]}
// litepack β 25 bytes. Same data:
40 08 CD 8F E5 B5 03 07 70 65 65 72 2D 78 31 05 01 16 CD 02 DC 22 A3 B2 03The schema lives in your code (or a shared JSON file), so key names never travel. Integers use variable-length encoding β small numbers cost 1 byte. Enums become indices. Flags pack into bits. 70β90% smaller than JSON on typical structured messages.
- π Real-time messaging β WebSocket / WebRTC / TCP protocols where messages are small and frequent (presence updates, game state, signaling, telemetry). This is where per-message overhead dominates.
- π± Mobile & metered connections β every byte is battery and quota.
- πΎ Compact storage β millions of small records in IndexedDB, Redis, or files.
- βοΈ Canonical bytes for signing β deterministic output, unlike
JSON.stringify(which depends on key order).
- Human-readable configs and logs β use JSON.
- Large repetitive payloads β litepack is compact encoding, not compression; run gzip/deflate on top (or instead) for multi-KB blobs.
- Schemaless data β if you truly can't know the shape ahead of time,
use the built-in
jsonfield type as an escape hatch, or CBOR/MsgPack.
- Installation Β· Quick Start Β· Complete example
- What the bytes look like Β· Types Β· API
- Streaming & chaining Β· Real-world examples
- Schema evolution Β· Robustness Β· Performance
- Common pitfalls Β· How does it compare?
npm i litepack
Or just drop litepack.js into your project β it's a single UMD file.
All of these work:
// Node.js / bundlers
var litepack = require('litepack');
import litepack from 'litepack';
// Browser β plain script tag β global
<script src="litepack.js"></script>
<script> litepack.encode(schema, data); </script>
// Service Worker
importScripts('/js/litepack.js'); // β self.litepackRuns in Node.js, browsers, Service Workers, Bun and Deno. TypeScript
declarations (litepack.d.ts) give you autocomplete even in plain-JS
projects β just keep the file next to litepack.js.
var litepack = require('litepack');
// 1. Define a schema β an array of [name, type] pairs
var userProto = [
['id', 'varint'],
['name', 'string'],
['email', 'string?'], // ? = optional, costs 0 bytes when absent
['age', 'uint8?']
];
// 2. Encode
var buf = litepack.encode(userProto, {
id: 12345, name: 'Aviv', email: 'aviv@example.com', age: 30
});
// β Uint8Array(29) (~70 bytes as JSON)
// 3. Decode
var user = litepack.decode(userProto, buf);
// β { id: 12345, name: 'Aviv', email: 'aviv@example.com', age: 30 }
// 4. Decoding bytes from the network? Use tryDecode β it never throws:
var msg = litepack.tryDecode(userProto, untrustedBytes);
if (!msg) return; // malformed / truncated / hostile input β droppedBoth sides share one schema module β that's the entire "protocol spec":
// ββ shared/protocol.js β the single source of truth ββββββββββββββ
var OP = { CHAT: 1, MOVE: 2 };
var chatProto = [
['op', 'const', 'uint8', OP.CHAT], // written & verified automatically
['from', 'varint'],
['text', 'string']
];
var moveProto = [
['op', 'const', 'uint8', OP.MOVE],
['id', 'varint'],
['x', 'svarint'],
['y', 'svarint']
];
// ββ sender (either side) βββββββββββββββββββββββββββββββββββββββββ
socket.send(litepack.encode(chatProto, { from: myId, text: 'hi!' }));
socket.send(litepack.encode(moveProto, { id: myId, x: -3, y: 12 }));
// ββ receiver (either side) βββββββββββββββββββββββββββββββββββββββ
socket.onmessage = function(event) {
var buf = new Uint8Array(event.data);
var op = buf[0]; // opcode is always byte 0
var msg = op === OP.CHAT ? litepack.tryDecode(chatProto, buf)
: op === OP.MOVE ? litepack.tryDecode(moveProto, buf)
: null;
if (!msg) return; // unknown op / malformed / hostile
if (op === OP.CHAT) showChat(msg.from, msg.text);
if (op === OP.MOVE) moveEntity(msg.id, msg.x, msg.y);
};That's the whole pattern: shared schemas, encode out, tryDecode in,
const validating the opcode for free. Everything else in this README is
refinement of this loop.
Understanding the wire format takes one example. Take this schema and data:
var proto = [
['id', 'varint'],
['flags', 'bits', [['active',1], ['mode',3]]],
['name', 'string'],
['email', 'string?']
];
litepack.encode(proto, { id: 300, name: 'Dan' }); // email omittedWire bytes: 00 AC 02 00 03 44 61 6E
| Bytes | Meaning |
|---|---|
00 |
optionals bitmask β 1 varint for ALL optional fields; email absent β bit 0 clear |
AC 02 |
id = 300 as varint (values 0β127 take 1 byte, 128β16383 take 2β¦) |
00 |
flags β 4 bits packed into 1 byte |
03 |
name length prefix (3) |
44 61 6E |
"Dan" in UTF-8 |
email costs zero bytes when absent. There are no field names, no
delimiters, no type tags β the schema on both sides is the contract.
Cheatsheet β every type at a glance:
| Type | Wire size | Use for |
|---|---|---|
uint8/16/32, int8/16/32 |
1/2/4 B fixed | values with a known range |
uint64, int64 |
8 B | file sizes, big ids (BigInt beyond 2^53) |
varint |
1-8 B | counts, ids, timestamps β usually small, sometimes big |
svarint |
1-8 B | signed deltas, coordinates |
timestamp |
= varint | readability alias |
float32/64 |
4/8 B | real numbers |
bool |
1 B | single flag (many flags β bits) |
string |
len + UTF-8 | text |
bytes |
len + raw | binary blobs |
fixed, uuid |
exactly N / 16 B | hashes, keys, ids β no prefix |
tail |
rest of buffer | final blob, top-level last field only |
enum |
1+ B index | one choice from a list |
set |
1+ B bitmask | multiple choices from a list (β€52) |
bits |
packed | several small ints in minimal bytes |
const |
= base type | protocol constants β auto-written, verified |
struct |
inline | nested object |
array |
count + items | lists, incl. arrays of structs |
map |
count + pairs | dynamic keys, typed values |
json |
len + JSON | free-form escape hatch |
| (custom) | len + codec | anything via litepack.codec() |
Any type + ? = optional (zero bytes when absent).
['level', 'uint8'] // 0..255 (1 byte)
['temp', 'int8'] // -128..127 (1 byte)
['port', 'uint16'] // 0..65535 (2 bytes)
['altitude', 'int16'] // -32768..32767 (2 bytes)
['color', 'uint32'] // 0..4294967295 (4 bytes)
['time', 'int32'] // signed 32-bit (4 bytes)
['filesize', 'uint64'] // 0..2^64 (8 bytes, BigInt above 2^53)
['balance', 'int64'] // signed 64-bit (8 bytes, BigInt beyond Β±2^53)
['count', 'varint'] // unsigned, variable size (1 byte for 0-127, 2 for 128-16383β¦)
['delta', 'svarint'] // SIGNED varint (zigzag) (-1 β 1 byte; small magnitudes stay small)Choosing between them: varint for counts, ids, timestamps β anything
usually small but occasionally big. Fixed-width types (uint32 etc.) when
the value is uniformly distributed (hashes, colors) β a varint would
cost more there. svarint for deltas and coordinates that go negative.
β οΈ varintis unsigned by design β encoding a negative value throws (usesvarint). Values above 2^53-1 also throw (useuint64).
['temperature', 'float32'] // 32-bit IEEE 754 (4 bytes)
['precise', 'float64'] // 64-bit IEEE 754 (8 bytes)
['active', 'bool'] // true/false (1 byte β pack many bools with 'bits' instead)['name', 'string'] // UTF-8 with varint length prefix
['payload', 'bytes'] // raw bytes with varint length prefix
['body', 'tail'] // last field ONLY β the rest of the buffer, no prefix
['hash', 'fixed', 32] // exactly 32 bytes, no length prefix (SHA-256, keys, UUIDs)fixed saves the length prefix when the size is known β a SHA-256 hash is
32 bytes on the wire, not 33. tail saves it for the final variable blob.
β οΈ On decode,bytes/fixed/tailreturn views into the input buffer (zero-copy, fast). If you reuse or mutate the network buffer after decoding, copy first:new Uint8Array(msg.payload)β or decode with{copy: true}.
The idea behind tail is one simple observation: your transport already
knows where the message ends. A WebSocket frame, a datagram, a
length-prefixed TCP frame β they all carry a message boundary. So a length
prefix on the final blob inside the message is redundant information.
tail deletes it: "everything from here to the end of the buffer".
Where it shines:
1. Envelope + opaque payload (tunneling). A relay forwards a message
without understanding its contents β it reads the header and passes the
tail along untouched:
var envelopeProto = [
['type', 'const', 'uint8', MSG.RELAY],
['fromPeer', 'varint'],
['toPeer', 'varint'],
['payload', 'tail'] // a complete inner message β not our business
];
// Layered protocols fall out naturally β the tail is itself decodable:
var outer = litepack.tryDecode(envelopeProto, wireBytes);
var inner = litepack.tryDecode(innerProto, outer.payload);2. Small header + big data (file/media chunks). The byte saved by the
missing prefix is nice; the real win is zero-copy β the decoded tail
is a view into the receive buffer, so a 16 KB chunk is never copied:
var chunkProto = [
['fileId', 'varint'],
['chunkIndex', 'varint'],
['data', 'tail']
];3. Signed raw bytes. header + the exact bytes that were signed β
tail hands them back byte-for-byte, untouched by any encoding.
bytes vs tail β the rule: use bytes when something comes after
it, or when it lives inside a struct or array item (a length is needed
to know where it ends). Use tail when it's the last thing in the message
and framing is the transport's job. That's also why the compiler enforces
"last field of a top-level schema only" β it's not an arbitrary limit,
it's the definition of when tail is meaningful at all. Inside nested
structs/arrays the compiler throws with a clear message pointing you to
bytes.
Put together, [const opcode] [header fields] [tail payload] is the
template for a near-perfect protocol message: every byte on the wire is
real information, zero structural overhead.
var msgProto = [
['type', 'const', 'uint8', 64], // written automatically, VERIFIED on decode
['id', 'varint']
];
litepack.encode(msgProto, { id: 5 }); // no need to pass `type` β the schema owns it
litepack.tryDecode(msgProto, buf); // returns null if byte 0 isn't 64Perfect for message-type codes and magic bytes: the constant disappears
from your data objects, and every decode validates it for free β a frame
with the wrong msgcode is rejected before you look at it. Any simple base
type works, including 'string' for protocol magic (['magic','const','string','LP1']).
['counters', 'map', 'string', 'varint'] // { views: 100, clicks: 7 }
['names', 'map', 'varint', 'string'] // numeric keys: { 1001: 'Aviv' }
['peers', 'map', 'varint', 'struct', [ // struct values
['name', 'string'], ['online', 'bool']
]]For when the keys aren't known ahead of time but the types are β metadata, per-name counters, id-indexed records. Wire format: varint count
- (key, value) pairs. Far more compact than the
jsontype ({a:1,b:2,c:3}β 10 bytes vs 20).
['createdAt', 'timestamp'] // readability alias for varint (ms epoch fits fine)
['requestId', 'uuid'] // exactly 16 bytes, no length prefix['meta', 'json'] // any JSON-serializable structureFor the part of a message that's genuinely free-form. Not compact β it's
JSON.stringify with a length prefix β but always available.
// Enum β single choice, stored as varint index
['status', 'enum', ['active', 'inactive', 'banned']]
// 'banned' β 1 byte on the wire (vs 8 bytes as a JSON string)
// Set β multiple choice, stored as one varint bitmask (like MySQL SET)
['perms', 'set', ['read', 'write', 'execute', 'admin']]
// ['read', 'execute'] β 1 byte (bits 0 + 2 = 0x05)Strictness rules that catch real bugs:
- Encoding a value not in the list throws β a typo like
'inActive'would otherwise silently become a different valid value (index 0). - A raw non-negative number passes through an enum untouched β forward compatibility for values defined by a newer peer. Decoding an out-of-range index likewise returns the raw number.
setsupports up to 52 options (varint bitmask limit) β more than that is a compile-time error, not silent corruption.
['flags', 'bits', [
['active', 1], // 1 bit β 0 or 1
['mode', 3], // 3 bits β 0 to 7
['quality', 4] // 4 bits β 0 to 15
]]
// 8 bits total = exactly 1 byte on the wire
litepack.encode(s, { flags: { active: 1, mode: 5, quality: 12 } });Perfect for sensor readings, permissions, and protocol flags. Subfield widths are 1β32 bits each; the total can exceed 32 bits β it packs correctly across as many bytes as needed.
['address', 'struct', [
['city', 'string'],
['zip', 'uint32']
]]
// Structs nest arbitrarily deep:
['order', 'struct', [
['id', 'varint'],
['customer', 'struct', [
['name', 'string'],
['address', 'struct', [['city', 'string'], ['zip', 'uint32']]]
]]
]]['scores', 'array', 'uint16'] // varint count prefix + items
['rgb', 'array', 'uint8', 3] // fixed count β no prefix, saves a byte
['tags', 'array', 'string']
['roles', 'array', 'enum', ['admin', 'user', 'guest']]
// Array of structs β fully supported, including nested arrays inside:
['streams', 'array', 'struct', [
['streamId', 'varint'],
['fromPeer', 'varint'],
['toPeers', 'array', 'varint']
]]
// Array of structs with fixed count
['points', 'array', 'struct', [['x','float32'], ['y','float32']], 2]There is no need to pre-encode struct entries into an array of
bytes β arrays of structs (with strings, optionals, enums, bits, and
nested arrays inside) work natively, and cost one length prefix less per
entry than the manual wrapping trick.
Different fields depending on a discriminator β one schema for a whole message family:
var msgProto = [
['seq', 'varint'],
['kind', 'uint8', {
'1': [['text', 'string']], // chat
'2': [['width', 'uint16'], ['height', 'uint16'], ['data', 'bytes']], // image
'3': [['lat', 'float64'], ['lon', 'float64']] // location
}]
];
litepack.encode(msgProto, { seq: 1, kind: 3, lat: 32.08, lon: 34.78 });
// decode reads `kind`, then parses only the matching branch β flat object outAny type takes a ? suffix. All optional fields in a struct share one
varint bitmask at the front β 8 optionals cost 1 byte of overhead total,
and each absent field costs nothing:
['email', 'string?']
['age', 'uint8?']
['tags', 'array?', 'string']Up to 52 optional fields per struct (compile-time error beyond β split into nested structs).
Default values β an options object as the last element of the def:
['retries', 'varint?', {default: 3}]
['tags', 'array?', 'string', {default: []}]On decode, an absent field gets the default instead of undefined (object
and array defaults are cloned per decode β no shared references). On
encode, absent still costs zero bytes. This is the painless way to evolve
schemas: add ['battery','uint8?',{default:100}] and every old message
decodes with a sensible value β no === undefined checks scattered
through your code.
Throws on invalid data: unknown enum/set values, negative varint,
values above 2^53-1, misplaced tail. Missing optional fields are free;
missing required scalars encode as zero-values (see assertComplete to
forbid that).
Accepts Uint8Array or ArrayBuffer. Throws on malformed input β
truncation, declared lengths exceeding the buffer, varint bombs, impossible
array counts β with the failing field named in the error. Trailing bytes
after the last field are ignored (that's what makes schema evolution work;
see below).
Pass {copy: true} as a third argument to detach decoded bytes /
fixed / uuid / tail views from the input buffer (one clone of the
input up front) β do this whenever you recycle network buffers:
var msg = litepack.decode(proto, reusableBuf, { copy: true });Same as decode, but returns null on malformed input instead of
throwing. The recommended call for network data:
var msg = litepack.tryDecode(proto, event.data);
if (!msg) return; // hostile/corrupt input handled, donevar Cert = litepack.compile(certProto);
var buf = Cert.encode(data);
var obj = Cert.decode(buf);Schemas used with encode/decode directly are compiled once and cached
via WeakMap β your schema array is never mutated and frozen schemas work.
compile() just makes call sites tidier. Either way: define schemas once
at module level and reuse the array β the cache is keyed by identity, so
re-parsing a schema from JSON on every call recompiles every time.
Exact encoded size without encoding β for frame budgets, preallocation, and "will this fit in one packet?" checks.
Throws if any non-optional field is undefined/null, recursing into
structs, with the full field path in the message. Essential before
producing canonical/signed bytes, where "missing field silently encodes as
empty" would let a broken record produce a valid-looking signature:
litepack.assertComplete(certSignProto, cert);
// β Error: litepack: missing required field 'publicKey'
var signBytes = litepack.encode(certSignProto, cert);For frames that carry more than one message: a shared header followed by a type-dependent body, or N messages packed into one send. See the dedicated section below for the full story.
litepack.writer() // chained-message builder
.write(schemaA, dataA) // queue a message (chainable)
.raw(preEncodedBytes) // splice raw bytes (signatures, blobs)
.byteLength() // total so far, without encoding
.bytes() // β one exactly-sized Uint8Array
var r = litepack.reader(buf); // sequential cursor
r.read(schema) // decode next message + advance (throws)
r.tryRead(schema) // null on malformed, cursor NOT advanced
r.peek(schema) // decode without advancing
r.skip(n) // jump over foreign bytes
r.remaining(); r.eof(); r.offset;
litepack.decodeFrom(schema, buf, offset) // β { value, bytesRead }
litepack.encodeInto(schema, data, buf, offset) // β bytes written (throws if no fit)
litepack.concat([buf1, buf2, ...]) // β one Uint8Array, single allocationRegister any encoder as a field type:
var cbor = require('cbor-x');
litepack.codec('cbor', {
encode: function(val) { return cbor.encode(val); },
decode: function(buf) { return cbor.decode(buf); }
});
var schema = [['id', 'uint32'], ['payload', 'cbor']];encode must be deterministic (same value β same bytes). litepack detects
a size mismatch between the sizing pass and the writing pass and throws,
rather than emitting a corrupted buffer. Provide estimateSize(val) to
skip the extra sizing encode for large values.
decode() answers "what is in this buffer". reader/writer answer
"what is next in this buffer" β which is what real protocols need.
Every message in your protocol starts the same way; the rest depends on the type. Instead of copying the header fields into every schema:
// Define the header ONCE
var headerProto = [
['type', 'uint8'],
['sub', 'uint8']
];
// Bodies are separate, small schemas
var scalarBody = [['subjectPeerId', 'varint'], ['value', 'varint']];
var listBody = [['subjectPeerId', 'varint'], ['value', 'array', 'varint']];
// ββ Sending ββ
var frame = litepack.writer()
.write(headerProto, { type: MSG.PEER_INFO, sub: SUB.BATTERY })
.write(scalarBody, { subjectPeerId: peerId, value: 88 })
.bytes();
// ββ Receiving ββ
var r = litepack.reader(frame);
var head = r.read(headerProto); // decoded, not byte-peeked
if (head.type !== MSG.PEER_INFO) return;
var body = r.read(head.sub === SUB.LIST ? listBody : scalarBody);Wire guarantee: chaining headerProto then bodyProto produces
byte-identical output to a single merged schema with the same fields in
the same order (as long as each part keeps its own optional fields). And
since schemas are plain arrays, headerProto.concat(scalarBody) is that
merged schema β old code using merged protos and new code using chained
reads interoperate freely.
Sending 50 tiny WebSocket messages costs 50 frames of transport overhead. Pack them:
var updateProto = [['entityId', 'varint'], ['x', 'svarint'], ['y', 'svarint']];
// sender β one allocation, one send
var w = litepack.writer();
for (var i = 0; i < updates.length; i++) w.write(updateProto, updates[i]);
socket.send(w.bytes());
// receiver β read until the frame is exhausted
var r = litepack.reader(event.data);
while (!r.eof()) {
var u = r.tryRead(updateProto);
if (!u) break; // malformed remainder β stop cleanly
applyUpdate(u);
}Over raw sockets, bytes arrive in arbitrary chunks β a message may be
split across packets. Decode errors caused by truncation carry
err.truncated === true, meaning "more bytes might complete this";
malformed errors don't, meaning "drop it":
var acc = new Uint8Array(0);
socket.on('data', function(chunk) {
acc = litepack.concat([acc, chunk]);
var offset = 0;
while (offset < acc.length) {
try {
var r = litepack.decodeFrom(msgProto, acc, offset);
handleMessage(r.value);
offset += r.bytesRead;
} catch (e) {
if (e.truncated) break; // partial message β wait for more bytes
throw e; // malformed β protocol error, disconnect
}
}
acc = acc.subarray(offset); // keep only the unconsumed remainder
if (acc.length > MAX_FRAME) socket.destroy(); // cap growth β don't let a
}); // hostile peer buffer foreverWhen you need manual control (index building, seeking through a file of packed records):
var rec = litepack.decodeFrom(recordProto, fileBytes, offset);
// rec.value β the decoded object
// rec.bytesRead β where the next record starts
offset += rec.bytesRead;A peer announces its battery level to the swarm. As JSON:
{"type":64,"sub":6,"subjectPeerId":918273645,"value":88} β 57 bytes.
var peerInfoProto = [
['type', 'uint8'],
['sub', 'uint8'],
['subjectPeerId', 'varint'],
['value', 'varint']
];
litepack.encode(peerInfoProto, { type:64, sub:6, subjectPeerId:918273645, value:88 });
// β 8 bytes (86% smaller). At 10 updates/sec Γ 50 peers, that's the
// difference between 28 KB/s and 4 KB/s of presence traffic.var stateProto = [
['tick', 'varint'],
['players', 'array', 'struct', [
['id', 'varint'],
['x', 'svarint'], // deltas go negative β svarint keeps them 1-2 bytes
['y', 'svarint'],
['flags', 'bits', [['alive',1], ['team',2], ['weapon',4], ['anim',5]]]
]]
];
// 8 players β 45 bytes/tick vs ~450 as JSON. At 30 ticks/sec: 1.3 KB/s vs 13 KB/s.var batchProto = [
['deviceId', 'uint32'],
['baseTime', 'varint'],
['readings', 'array', 'struct', [
['dt', 'varint'], // ms since previous reading β tiny varints
['temp', 'svarint'], // tenths of a degree, signed
['state', 'bits', [['ok',1], ['battery',7]]]
]],
['signature', 'fixed', 64] // exactly 64 bytes, no prefix
];JSON.stringify output depends on key insertion order β the same cert can
hash differently on the signer and the verifier. litepack output depends
only on the schema:
var certSignProto = [
['version', 'uint8'],
['timestamp', 'varint'],
['issuerId', 'varint'],
['publicKey', 'bytes'],
['subjectId', 'bytes']
];
litepack.assertComplete(certSignProto, cert); // refuse to sign a hole
var signBytes = litepack.encode(certSignProto, cert);
var signature = await sign(sha256(signBytes));
// The verifier rebuilds signBytes from the received cert with the SAME
// schema β guaranteed identical bytes.var wireProto = [
['msgType', 'uint8', {
'1': [['roomId', 'varint'], ['text', 'string']],
'2': [['roomId', 'varint'], ['imageId', 'varint'], ['thumb', 'bytes']],
'3': [['peerId', 'varint']] // typing indicator
}]
];
var msg = litepack.tryDecode(wireProto, packet);
if (!msg) return;
switch (msg.msgType) { /* fields of the matching branch are on msg */ }Protocols change. The rules for staying compatible are simple and mechanical:
β Safe β add optional fields at the END:
// v1 // v2
[['id','varint'], [['id','varint'],
['name','string']] ['name','string'],
['battery','uint8?']] // β new, optional, last- Old decoder + new message β reads its known fields, ignores the trailing bytes. Works.
- New decoder + old message β
batterybit absent in the bitmask βundefined. Works.
β Breaking β never do these to a live protocol:
- Insert/remove/reorder fields anywhere but the end.
- Change a field's type.
- Add a required field (old messages will fail to decode as truncated).
- Reorder
enum/setoption lists (indices are the wire values β append new options at the end only).
For bigger changes, bump a version discriminator and use variants to carry both shapes during the transition.
Two trust levels, by design:
| Direction | Policy | Why |
|---|---|---|
| encode (your data) | strict β throws on typos, negatives, overflow, misplaced tail |
a bug in your code should explode in dev, not ship corrupted bytes |
| decode (their data) | defensive β bounded, validated; throws (or tryDecode β null) on anything malformed |
wire input is attacker-controlled; garbage in must never mean garbage out, OOM, or a CPU spin |
Decode hard limits: varints capped at 8 bytes; array counts validated against remaining buffer bytes before allocation; every declared length checked before use; truncation detected per-field with the field name in the error.
Measured on Node 22 (1M iterations, typical small messages):
| Operation | Throughput |
|---|---|
| encode, mixed message (ints + string + array) | ~1.2M ops/s |
| decode, same | ~3.4M ops/s |
| decode, enum+set message | ~7.5M ops/s |
| encode, 5 structs w/ nested arrays | ~400K ops/s |
| chained writer (2 parts) | no penalty vs merged schema |
Under the hood: schemas compile once to closures (cached by identity),
string sizing scans without allocating, UTF-8 encodes straight into the
output buffer (TextEncoder.encodeInto), enum/set lookups are prebuilt
maps, buffers are allocated exactly once per encode.
| Data | JSON | litepack | Saved |
|---|---|---|---|
{ id: 123, name: 'Aviv' } |
27 bytes | 6 bytes | 78% |
| User with enum + set + array | 123 bytes | 26 bytes | 79% |
| Sensor with bitfields + tail | 95 bytes | 13 bytes | 86% |
| 5 stream-structs with nested arrays | 282 bytes | 39 bytes | 86% |
Schemas are plain arrays β store them in JSON files, share between server and client:
[
["id", "varint"],
["name", "string"],
["status", "enum", ["active", "inactive", "banned"]],
["streams", "array", "struct", [["streamId", "varint"], ["toPeers", "array", "varint"]]]
]var schema = JSON.parse(fs.readFileSync('schema.json')); // parse ONCE
litepack.encode(schema, data); // reuse the arrayThe wire format is unchanged β 1.0 and 1.1 peers interoperate freely. Behavior changes to be aware of:
decodenow throws on malformed/truncated input (previously returned silently wrong data). Wrap untrusted decodes withtryDecode.encodenow throws on unknown enum/set values, negativevarint, andvarintabove 2^53-1 (previously silently encoded a different value).- More than 31 optional fields / set options previously corrupted data silently; now supported correctly up to 52, compile-time error beyond.
- Bitfields totalling more than 32 bits previously corrupted silently; now packed correctly.
tailoutside the last top-level position is now a compile-time error (previously corrupted silently).- New types:
svarint,int64,json,const,map,uuid,timestamp. New API:compile,tryDecode,byteLength,assertComplete,reader,writer,decodeFrom,encodeInto,concat,{copy:true}decode option,{default:β¦}field option,err.truncatedfor stream parsing.
litepack.js β single file, UMD, zero dependencies
litepack.d.ts β TypeScript declarations (editor autocomplete)
litepack.test.js β core test suite
litepack.chain.test.js β streaming/composition test suite
litepack-tool.html β interactive schema builder & tester
The five mistakes everyone makes once β read this and make zero:
1. Field order IS the protocol. There are no field names or tags on the
wire. If the sender's schema says [id, name] and the receiver's says
[name, id], you get garbage β both sides must use the identical schema.
Share it as a module or JSON file; never retype it.
2. Decoded bytes/fixed/uuid/tail are views. They point into
the input buffer. If you recycle that buffer (common with sockets), your
decoded values silently change. Recycling? β decode(proto, buf, {copy:true}).
3. Reparsing schemas kills the cache. Compilation is cached by array
identity. JSON.parse(schemaText) inside a hot loop creates a new array
every call β recompiles every call. Parse once at module level.
4. varint is unsigned. -1 throws (on purpose). Deltas and
coordinates that go negative β svarint.
5. Untrusted input goes through tryDecode. Plain decode throws on
malformed bytes β correct for your own data, but an unhandled exception in
a socket handler if a hostile peer sends garbage. tryDecode β null β
return.
An honest map of the landscape:
| Approach | Strong at | Weak at (vs litepack) | |
|---|---|---|---|
| litepack | schema in code, single file | tiny messages, zero build step, JSON-storable schemas, bits/set/tail/variants, deterministic bytes |
ecosystem size, cross-language ports |
| Protocol Buffers | .proto files + codegen |
huge ecosystem, every language, field-tag evolution (can delete fields) | heavy toolchain; requires build step or a large runtime; no bitfields |
| MessagePack / CBOR | schemaless | zero setup, any shape | sends key names in every message β 2-3Γ larger on small structured messages |
| FlatBuffers | schema + zero-copy access | reading giant buffers without deserializing | complex API, codegen required, larger output for small messages |
| Avro | schema as JSON | schema resolution between versions, big-data tooling | JVM-centric ecosystem, heavier runtime |
Where litepack wins: small, frequent, structured messages β the per-message overhead of key names (MessagePack) or toolchain (protobuf) dominates exactly there. One file, no build step, runs in a Service Worker, schemas are plain arrays you can ship as JSON, and the output is deterministic enough to sign.
Where to pick something else: a public API consumed by parties you don't control (protobuf's tag-based evolution is more forgiving), truly schemaless payloads (CBOR), or random access into multi-megabyte buffers (FlatBuffers).
Pull requests are welcome! Please open an issue before submitting major changes.
MIT