Datassert is a high-performance CLI for building a DuckDB-backed assertion store from NCATS Translator BABEL export files, with a focus on fast local builds and simple command-driven workflows.

# Install CLI from GitHub
go install github.com/SkyeAv/datassert@latest

# Verify install
datassert --help

# Build a Datassert database (downloads BABEL data automatically)
datassert build

The build command automatically downloads BABEL exports from RENCI (https://stars.renci.org/var/babel_outputs), processes them, and produces sharded DuckDB databases.

1. Download -- BABEL class and synonym files are downloaded from RENCI and split into LZ4-compressed NDJSON chunks under ./datassert/downloads/classes/ and ./datassert/downloads/synonyms/.
2. Lookup -- Class files (*.ndjson.lz4) are read to build an in-memory equivalent-identifier lookup.
3. Parquet Staging -- Synonym files are processed with the lookup, quality-controlled, and written as sharded Parquet files to ./datassert/parquets/.
4. DuckDB Generation -- Parquet files are loaded into 10 sharded DuckDB databases under ./datassert/data/.

• 10 sharded DuckDB databases are written to ./datassert/data/{0..9}.duckdb.
• Each shard contains SOURCES, CATEGORIES, CURIES, and SYNONYMS tables, deduplicated, sorted, and indexed for query performance.
• Staging Parquet files are written to ./datassert/parquets/{0..9}/.

# Full build (download, process, and generate databases)
datassert build

# Skip downloads if BABEL files were already fetched
datassert build --skip-downloads

# Rebuild DuckDB databases from existing Parquet files
datassert build --use-existing-parquets

• Displays progress bars for download, class lookup, synonym processing, and DuckDB build phases.
• Uses 90% of available CPUs for concurrent processing.
• Downloads are retried up to 5 times on failure with a 5-second backoff.
• All working files are stored under ./datassert/.

Skye Lane Goetz