BrewFS: High-performance distributed storage, built in Rust.

Install · Benchmarks · Architecture · Documentation · 中文

BrewFS is an independent distributed filesystem for container, AI, and object-storage-heavy workloads. It combines a POSIX-like FUSE interface with pluggable transactional metadata and S3-compatible data storage.

RustFS is one of the S3-compatible object storage backends supported by BrewFS and is used in the repository's reproducible benchmark and filesystem test profiles.

The July 24, 2026 Redis and TiKV comparison uses matched buffered fio profiles, preserved local read caches, and strict post-write drain. PERF_FIO_SIZE=512m is the per-job size: time-based workloads run for 20 seconds, while Large read is a one-pass 8-job, 4 GiB aggregate workload. Write results are reported as complete end-to-end throughput because foreground fio bandwidth alone omits queued writeback work.

• Fast data path: chunked I/O, memory and SSD caches, read-ahead, writeback, and large-write coalescing.
• Rust throughout: one modern, memory-safe implementation from FUSE and VFS to metadata and object storage.
• Storage freedom: Redis, TiKV, etcd, PostgreSQL, or SQLite metadata with S3-compatible or local object data.
• Operationally testable: xfstests, pjdfstest, LTP, stress-ng, fio, metadata benchmarks, fuzzing, and Docker Compose runners live in the repository.

This July 24, 2026 snapshot compares BrewFS with JuiceFS 1.3.1 using Redis and TiKV metadata plus RustFS S3-compatible object storage. The four complete runs used buffered io_uring, 4 MiB I/O, a 512 MiB per-job fio size, disabled compression, durable read prefill, cache-preserving remounts, and explicit post-write drain. Time-based fio workloads ran for 20 seconds; Large read and Large write transferred 4 GiB with eight jobs. All ten tools passed in every run and every write drain ended with zero queued bytes.

No single number describes writeback filesystems correctly. The first table is application-visible throughput, based on bytes divided by complete tool wall time. The second extends the finish line through background writeback drain. Across 14 application-visible data rows and 10 metadata rows, BrewFS leads 15 of 24: 10 of 14 data rows and 5 of 10 metadata rows. Stable persistent-cache Large read is effectively tied; JuiceFS still leads sequential read, strictly drained pure writes, and TiKV create/open/rename.

Eight BrewFS Redis upload workers were rejected because mixed random I/O fell about 23% during fio and about 12% after drain. An aggressive 4 GiB read plus 4 GiB write TiKV profile was rejected after it caused multi-minute close tails. The published profiles are the fastest accepted settings that preserved the mixed-I/O gate and host stability.

Actual bytes divided by complete fio process wall time. Mixed random I/O is read plus write throughput.

Read rows use durable prefill followed by an unmount/remount that preserves the local SSD cache. They are persistent-cache results, not cold object-store results. Large read uses three remount/read rounds, targeted POSIX_FADV_DONTNEED eviction of each filesystem's local cache files, and the median bandwidth. This prevents an asymmetric host page-cache hit from being reported as a persistent-cache advantage. All four stable Large read artifacts recorded zero object GET bytes during the measured phase.

This separate diagnostic measures the fully hot local page-cache path, not durable SSD-cache throughput and not cold object-store download. After the same durable prefill and cache-preserving remount, each filesystem performs five unreported 4 GiB warmup passes and then three measured 4 GiB passes in the same mount. No local-cache eviction or inter-pass remount occurs. Both systems issued zero object GET bytes during the measured phase.

The small spread makes this a repeatable host-memory-cache result. It is published separately so it cannot be mistaken for the remount-stable persistent-cache comparison above. BrewFS's promoted-slice cache is accounted for by the zero S3 GET evidence; its generic block-cache hit counter does not include that path.

Actual bytes divided by active I/O time plus post-write drain. This exposes work left on local SSD after the application finishes. Mixed random I/O reports total read plus write bytes.

BrewFS's SSD writeback cache is working: it shortens application-visible pure write time and strongly improves mixed I/O. The remaining pure-write deficit appears after the foreground phase, where object upload and metadata commit costs dominate. Large-write P99 close/fsync tails remain an optimization target rather than being hidden behind foreground bandwidth.

Each cell is tool wall / post-drain. Tool wall includes startup, active I/O, close, and fsync; post-drain ends only when the queue is empty.

Operations per second from the same complete metaperf runs.

These are all ten tools from each accepted run, not selected microbenchmarks.

Complete-matrix artifacts:

• BrewFS Redis: perf-run-1784876227-9934
• JuiceFS Redis: juicefs-perf-run-1784879992-17249
• BrewFS TiKV: perf-run-1784878850-2470
• JuiceFS TiKV: juicefs-perf-run-1784880709-16460

Stable Large read artifacts:

• BrewFS Redis: perf-run-1784889951-27900
• JuiceFS Redis: juicefs-perf-run-1784890100-1138

Hot local-cache Large read artifacts (Redis, five warmups and three measured passes):

• BrewFS Redis: perf-run-1784894177-23661
• JuiceFS Redis: juicefs-perf-run-1784894284-19194
• BrewFS TiKV: perf-run-1784890275-8226
• JuiceFS TiKV: juicefs-perf-run-1784890444-29768

Each artifact contains effective settings, raw fio JSON, tool logs, cache/object counters, drain samples, warnings, and a generated report under docker/compose-xfstests/artifacts/<run>/. These are single-host engineering results, not universal deployment claims.

PERF_LOG_TO_CONSOLE=false PERF_FIO_SIZE=512m PERF_FIO_RUNTIME=20 \
  bash docker/compose-xfstests/run_redis_perf.sh --s3 \
  --writeback-throughput-profile \
  --tools "fio-seqread fio-seqwrite fio-randread fio-randwrite fio-randrw fio-bigread fio-bigwrite metaperf dirstress dirperf"

JUICEFS_META_BACKEND=redis PERF_LOG_TO_CONSOLE=false \
PERF_FIO_SIZE=512m PERF_FIO_RUNTIME=20 \
  bash docker/compose-xfstests/run_juicefs_perf.sh \
  --writeback-throughput-profile \
  --tools "fio-seqread fio-seqwrite fio-randread fio-randwrite fio-randrw fio-bigread fio-bigwrite metaperf dirstress dirperf"

Use run_tikv_perf.sh for BrewFS TiKV and set JUICEFS_META_BACKEND=tikv for the JuiceFS TiKV run.

Install a complete single-node Linux stack with Redis, RustFS, systemd, and a BrewFS FUSE mount:

curl -fsSL https://raw.githubusercontent.com/brewfs/brewfs/main/scripts/install_brewfs_single_node.sh \
  | sudo bash -s -- install

Or build from source with Rust 1.85+ and fuse3:

cargo build -p brewfs --release

mkdir -p /tmp/brewfs-mnt /tmp/brewfs-data
target/release/brewfs mount /tmp/brewfs-mnt \
  --data-backend local-fs \
  --data-dir /tmp/brewfs-data \
  --meta-backend sqlx \
  --meta-url sqlite:///tmp/brewfs-meta.db

See the binary deployment guide and configuration reference for production backends, tuning profiles, upgrades, and uninstall steps.

BrewFS separates the filesystem interface, metadata, and object data paths:

• FUSE + VFS provide inode-based POSIX operations.
• Metadata stores namespaces, attributes, slices, sessions, and transactions in Redis, TiKV, etcd, PostgreSQL, or SQLite.
• Chunk + cache map files into 64 MiB chunks and 4 MiB blocks, with memory/SSD caching, compaction, and garbage collection.
• Object adapters persist blocks to S3-compatible systems such as RustFS, MinIO, AWS S3, and Ceph RGW, or to local storage.

Core operations include create, read, write, truncate, sparse files, rename, hardlinks, symlinks, byte-range locks, compaction, delayed deletion, and runtime info/gc control commands.

cargo test -p brewfs

cd docker
bash compose-xfstests/run_redis_xfstests.sh --cases "generic/001"
bash compose-xfstests/run_redis_pjdfstest.sh

The testing guide covers Redis, TiKV, RustFS, xfstests, pjdfstest, LTP, stress-ng, and performance runners.

• Documentation index
• Architecture
• Configuration
• Binary deployment
• Benchmark guide
• BrewFS/JuiceFS gap analysis

Issues and pull requests are welcome. Please keep behavior changes, tests, and documentation together whenever possible.

BrewFS treats filesystem correctness as a release requirement, not a best-effort compatibility claim. Its FUSE and metadata paths are continuously exercised with xfstests, pjdfstest, the Linux Test Project (LTP), and stress-ng across SQLite, Redis, etcd, and TiKV metadata backends.

The current validation baseline completes all 708 configured xfstests cases on every supported metadata backend. Redis and TiKV also pass the complete pjdfstest corpus: 246 test files and 9,134 assertions with no default exclusions. The configured LTP filesystem profiles pass on all four backends. Kernel and FUSE limitations that cannot yet be implemented reliably are narrowly excluded, documented with reproducible evidence, and kept visible for future revalidation.

This breadth of testing gives BrewFS a strong POSIX correctness foundation for builds, containers, AI pipelines, and other workloads that depend on predictable filesystem behavior. See the filesystem test suite matrix for current artifacts, coverage, and known limitations.

Questions, deployment discussions, and collaboration inquiries are welcome at genedna@gmail.com.