Jido.VFS is a filesystem abstraction for Elixir providing a unified interface over many storage backends. It allows you to swap out filesystems on the fly without needing to rewrite your application code. Eliminate vendor lock-in, reduce technical debt, and improve testability.
- Unified API - Same operations work across all adapters
- Multiple Backends - Local, S3, Sprite, Git, GitHub, ETS, and InMemory storage
- Version Control - Git, Sprite, ETS, and InMemory adapters support versioning
- Streaming - Efficient handling of large files
- Cross-Filesystem Operations - Copy files between different storage backends
- Visibility Controls - Public/private file permissions
- Public API return shape is deterministic:
:ok,{:ok, value}, or{:error, %Jido.VFS.Errors.*{}} - Unsupported operations always return
%Jido.VFS.Errors.UnsupportedOperation{operation, adapter} - Paths are normalized before adapter calls and traversal/absolute paths are rejected with typed errors
- Cross-filesystem copy is memory-bounded: native copy if available, then streaming, then tempfile spooling fallback
- Versioned adapters (
Git,ETS,InMemory) return%Jido.VFS.Revision{}values fromrevisions/3
| Adapter | Use Case | Streaming | Versioning | Notes |
|---|---|---|---|---|
Jido.VFS.Adapter.Local |
Local filesystem | Yes | No | Full local filesystem operations |
Jido.VFS.Adapter.S3 |
AWS S3 / Minio | Yes | No | Prefix-scoped clear/listing, multipart stream uploads |
Jido.VFS.Adapter.Sprite |
Fly.io Sprites | Yes | Yes | Shell-command filesystem + checkpoint-backed versioning |
Jido.VFS.Adapter.Git |
Git repositories | Yes | Yes | Deterministic bootstrap commit, rollback support |
Jido.VFS.Adapter.GitHub |
GitHub API | No | No | Remote read/write via API, typed unsupported ops |
Jido.VFS.Adapter.ETS |
ETS tables | Yes | Yes | Version storage hardened without dynamic atoms |
Jido.VFS.Adapter.InMemory |
Testing | Yes | Yes | Ephemeral storage with version snapshots |
Use supports?/2 before optional operations:
if Jido.VFS.supports?(filesystem, :write_stream) do
{:ok, stream} = Jido.VFS.write_stream(filesystem, "large.bin")
Stream.into(data_stream, stream) |> Stream.run()
else
:ok = Jido.VFS.write(filesystem, "large.bin", IO.iodata_to_binary(Enum.to_list(data_stream)))
endThis avoids relying on matching adapter error payloads to determine capabilities.
# Direct filesystem configuration
filesystem = Jido.VFS.Adapter.Local.configure(prefix: "/home/user/storage")
# Write and read files
:ok = Jido.VFS.write(filesystem, "test.txt", "Hello World")
{:ok, "Hello World"} = Jido.VFS.read(filesystem, "test.txt")
# Module-based filesystem (recommended for reuse)
defmodule MyStorage do
use Jido.VFS.Filesystem,
adapter: Jido.VFS.Adapter.Local,
prefix: "/home/user/storage"
end
MyStorage.write("test.txt", "Hello World")
{:ok, "Hello World"} = MyStorage.read("test.txt")The Local adapter provides standard filesystem operations:
filesystem = Jido.VFS.Adapter.Local.configure(prefix: "/path/to/storage")
# Basic operations
:ok = Jido.VFS.write(filesystem, "file.txt", "content")
{:ok, content} = Jido.VFS.read(filesystem, "file.txt")
:ok = Jido.VFS.delete(filesystem, "file.txt")
# Copy and move
:ok = Jido.VFS.copy(filesystem, "source.txt", "dest.txt")
:ok = Jido.VFS.move(filesystem, "old.txt", "new.txt")
# Directory operations
:ok = Jido.VFS.create_directory(filesystem, "new-folder")
{:ok, entries} = Jido.VFS.list_contents(filesystem, "folder/")
:ok = Jido.VFS.delete_directory(filesystem, "old-folder")
# File info
{:ok, stat} = Jido.VFS.stat(filesystem, "file.txt")
{:ok, :exists} = Jido.VFS.file_exists(filesystem, "file.txt")The S3 adapter works with AWS S3, Minio, and S3-compatible storage:
# Configure S3 filesystem
filesystem = Jido.VFS.Adapter.S3.configure(
bucket: "my-bucket",
prefix: "uploads/",
region: "us-east-1"
)
# For Minio or custom S3-compatible storage
filesystem = Jido.VFS.Adapter.S3.configure(
bucket: "my-bucket",
host: "localhost",
port: 9000,
scheme: "http://",
access_key_id: "minioadmin",
secret_access_key: "minioadmin"
)
# All standard operations work
:ok = Jido.VFS.write(filesystem, "document.pdf", pdf_binary)
{:ok, content} = Jido.VFS.read(filesystem, "document.pdf")
# Streaming for large files
{:ok, stream} = Jido.VFS.read_stream(filesystem, "large-file.bin", chunk_size: 65536)
Enum.each(stream, fn chunk -> process(chunk) end)The Sprite adapter executes shell commands on a Fly.io Sprite
through sprites-ex.
# Add sprites-ex in your project if needed
# {:sprites, github: "superfly/sprites-ex"}
filesystem =
Jido.VFS.Adapter.Sprite.configure(
sprite_name: "my-sprite",
token: System.fetch_env!("SPRITES_TOKEN"),
root: "/workspace",
encoding: :base64
)
:ok = Jido.VFS.write(filesystem, "notes/hello.txt", "hello sprite")
{:ok, "hello sprite"} = Jido.VFS.read(filesystem, "notes/hello.txt")
{:ok, entries} = Jido.VFS.list_contents(filesystem, "notes")encoding: :base64is binary-safe and default.encoding: :rawis text-oriented and avoids base64 overhead.- Pass
create_on_demand: trueto call Sprite create during configure.
Sprite versioning is backed by Sprite checkpoints:
:ok = Jido.VFS.write(filesystem, "doc.txt", "v1")
:ok = Jido.VFS.commit(filesystem, "checkpoint v1")
:ok = Jido.VFS.write(filesystem, "doc.txt", "v2")
:ok = Jido.VFS.commit(filesystem, "checkpoint v2")
{:ok, revisions} = Jido.VFS.revisions(filesystem, "doc.txt")
old_revision = List.last(revisions)
{:ok, "v1"} = Jido.VFS.read_revision(filesystem, "doc.txt", old_revision.sha)
:ok = Jido.VFS.rollback(filesystem, old_revision.sha, path: "doc.txt")The Git adapter provides version-controlled filesystem operations:
# Manual commit mode - you control when commits happen
filesystem = Jido.VFS.Adapter.Git.configure(
path: "/path/to/repo",
mode: :manual,
author: [name: "Bot", email: "bot@example.com"]
)
# Write files and commit manually
Jido.VFS.write(filesystem, "document.txt", "Version 1")
Jido.VFS.write(filesystem, "notes.txt", "Some notes")
:ok = Jido.VFS.commit(filesystem, "Add initial documents")
# Auto-commit mode - each write creates a commit
filesystem = Jido.VFS.Adapter.Git.configure(
path: "/path/to/repo",
mode: :auto
)
Jido.VFS.write(filesystem, "file.txt", "content") # Automatically committed
# View revision history
{:ok, revisions} = Jido.VFS.revisions(filesystem, "document.txt")
# Read historical versions
{:ok, old_content} = Jido.VFS.read_revision(filesystem, "document.txt", revision_sha)
# Rollback to a previous revision
:ok = Jido.VFS.rollback(filesystem, revision_sha)The GitHub adapter allows you to interact with GitHub repositories as a virtual filesystem:
# Read-only access to public repos
filesystem = Jido.VFS.Adapter.GitHub.configure(
owner: "octocat",
repo: "Hello-World",
ref: "main"
)
{:ok, content} = Jido.VFS.read(filesystem, "README.md")
{:ok, files} = Jido.VFS.list_contents(filesystem, "src/")
# Authenticated access for write operations
filesystem = Jido.VFS.Adapter.GitHub.configure(
owner: "your-username",
repo: "your-repo",
ref: "main",
auth: %{access_token: "ghp_your_token"},
commit_info: %{
message: "Update via Jido.VFS",
committer: %{name: "Your Name", email: "you@example.com"},
author: %{name: "Your Name", email: "you@example.com"}
}
)
# Write files (creates commits)
Jido.VFS.write(filesystem, "new_file.txt", "Hello GitHub!",
message: "Add new file via Jido.VFS")These adapters are ideal for testing and caching:
# ETS adapter - persists to ETS table
filesystem = Jido.VFS.Adapter.ETS.configure(name: :my_cache)
# InMemory adapter - ephemeral storage
filesystem = Jido.VFS.Adapter.InMemory.configure(name: :test_fs)
# Both support versioning
Jido.VFS.write(filesystem, "file.txt", "v1")
:ok = Jido.VFS.commit(filesystem, "Version 1")
Jido.VFS.write(filesystem, "file.txt", "v2")
:ok = Jido.VFS.commit(filesystem, "Version 2")
{:ok, revisions} = Jido.VFS.revisions(filesystem, "file.txt")
{:ok, "v1"} = Jido.VFS.read_revision(filesystem, "file.txt", first_revision_id)Copy files between different storage backends:
local_fs = Jido.VFS.Adapter.Local.configure(prefix: "/local/storage")
s3_fs = Jido.VFS.Adapter.S3.configure(bucket: "my-bucket")
# Copy from local to S3
:ok = Jido.VFS.copy_between_filesystem(
{local_fs, "document.pdf"},
{s3_fs, "uploads/document.pdf"}
)
# Copy from S3 to local
:ok = Jido.VFS.copy_between_filesystem(
{s3_fs, "backup.zip"},
{local_fs, "downloads/backup.zip"}
)copy_between_filesystem/3 prefers native adapter copy, then stream-based copy, and finally tempfile spooling for bounded-memory fallback behavior.
Efficiently handle large files with streaming:
# Read stream
{:ok, stream} = Jido.VFS.read_stream(filesystem, "large-file.bin", chunk_size: 65536)
Enum.each(stream, fn chunk -> process(chunk) end)
# Write stream
{:ok, stream} = Jido.VFS.write_stream(filesystem, "output.bin")
data |> Stream.into(stream) |> Stream.run()Control file permissions with visibility settings:
# Write with visibility
:ok = Jido.VFS.write(filesystem, "public-file.txt", "content", visibility: :public)
:ok = Jido.VFS.write(filesystem, "private-file.txt", "secret", visibility: :private)
# Get/set visibility
{:ok, :public} = Jido.VFS.visibility(filesystem, "public-file.txt")
:ok = Jido.VFS.set_visibility(filesystem, "file.txt", :private)If your project has Igniter available, you can install Jido VFS using the command
mix igniter.install jido_vfsAdd jido_vfs to your list of dependencies in mix.exs:
def deps do
[
{:jido_vfs, "~> 1.0"}
]
endFull documentation is available at HexDocs.
Additional guides:
Adapter authors should follow the Adapter Onboarding Checklist to ensure metadata callbacks, typed errors, and contract coverage are complete.
Apache-2.0 - see LICENSE.md for details.
jido_vfs is the filesystem abstraction layer for Jido packages, with adapters for local and remote storage backends.
For Jido agents, prefer wrapping jido_vfs with Jido.Action modules in a higher-level package or application rather than adding jido_action as a required dependency here. See the
Jido Agent Integration guide.
- Unit/adapter tests:
mix test - Full quality gate:
mix quality - Release preflight:
mix release.ready - Optional flaky cases:
mix test --include flaky