local: per-checkout stacks — every checkout runs its own isolated stack - #3137
#3137local: per-checkout stacks — every checkout runs its own isolated stack#3137jgraettinger merged 7 commits intomasterestuary/flow:masterfrom johnny/multi-worktree-infrastructure-da0902estuary/flow:johnny/multi-worktree-infrastructure-da0902Copy head branch name to clipboard
Conversation
995bddb to
6b0868e
Compare
skord
left a comment
There was a problem hiding this comment.
A couple of non-blocking observations on the per-checkout local tooling. Both are about failure modes rather than the happy path, which works well. Details inline.
| // local anon api_key default). Local-stack ports are dynamic, so a local | ||
| // deployment must pass API_ENDPOINT/AGENT_ENDPOINT explicitly — the | ||
| // local:data-plane task does this in the generated dekaf env file. | ||
| let (api_endpoint, api_key) = (cli.api_endpoint, cli.api_key); |
There was a problem hiding this comment.
dekaf --local with no endpoint now silently targets production.
Dropping the default_value_if("local", ...) from api_endpoint (and the if cli.local override that used to live here) means --local still forces the local anon api_key, but api_endpoint falls through to DEFAULT_PG_URL, which is production. So a hand-run dekaf --local points a local anon token at the prod PostgREST endpoint, where on master it used LOCAL_PG_URL.
The managed path is fine: local:data-plane writes API_ENDPOINT/AGENT_ENDPOINT into the generated dekaf env file, so mise run local:stack --dekaf always has them. The gap is a bare dekaf --local (or any invocation outside that env file), which fails silently against prod instead of failing loud. That is at odds with the fail-loud stance the rest of this PR takes.
Suggested guard: if --local is set but the endpoints are still the prod defaults (never right for a local stack), bail with something actionable.
// `--local` selects the local anon api_key, but local-stack ports are dynamic
// so there is no compiled local endpoint default. Guard the surprising case of
// `--local` with no endpoint: clap would otherwise fall back to the *production*
// defaults, pointing a local token at prod. local:data-plane supplies these in
// dekaf's generated env file; a hand-run must pass them too.
if cli.local {
if cli.api_endpoint == *DEFAULT_PG_URL {
anyhow::bail!(
"--local requires an explicit API_ENDPOINT (or --api-endpoint): \
local-stack ports are dynamic, so there is no local default. Run \
dekaf via `mise run local:data-plane`, or set API_ENDPOINT to your \
stack's PostgREST URL (see `mise run local:stack-info`)."
);
}
if cli.agent_endpoint == *DEFAULT_AGENT_URL {
anyhow::bail!(
"--local requires an explicit AGENT_ENDPOINT (or --agent-endpoint): \
local-stack ports are dynamic, so there is no local default. Run \
dekaf via `mise run local:data-plane`, or set AGENT_ENDPOINT to your \
stack's agent URL (see `mise run local:stack-info`)."
);
}
}Comparing against the prod default Url is reliable here, since a real local stack's endpoint is never the prod URL, and passing the prod URL alongside --local is nonsensical anyway. (DEFAULT_PG_URL/DEFAULT_AGENT_URL are already imported.) The tidier but larger alternative is making both Option<Url> and resolving explicitly, but for a one-file dev-tooling guard the equality check is the smaller diff.
Non-blocking: the managed flow is unaffected; this closes a footgun for hand-run invocations.
| case "${_se_alloc_rc}" in | ||
| 0) IFS=$'\t' read -r _se_index _se_name _se_new <<<"${_se_alloc}" ;; | ||
| 2) _se_die "cannot lock ${_se_lock} — check permissions" ;; | ||
| 3) _se_die "no free stack index (0..15 all taken); run 'mise run local:stack-release' in an unused checkout" ;; |
There was a problem hiding this comment.
Stale registry entries pin their index forever, with no way to reclaim once the checkout is gone.
If a worktree is removed with a plain rm -rf (rather than local:stack-release), its line stays in stacks.tsv and its index stays claimed. After enough churn you hit this no free stack index (0..15) error, but the remedy it suggests (running stack-release in that checkout) is impossible because the checkout no longer exists. There is currently no escape hatch short of hand-editing stacks.tsv.
The reason this is not simply "reap entries whose root is missing during allocation" is a collision hazard: deleting a worktree's directory does not stop its systemctl --user units, which keep holding their ports. A naive auto-reap could free an index and hand it to a new checkout while the old stack's services are still live, which is exactly the port collision the sticky-registry design avoids. Any reap has to gate on root gone AND units idle.
Suggested: a dedicated local:stack-prune task that reaps dead-root slots (and their artifacts, guarded like stack-release) while keeping any entry whose services are still active. And point this error message at it:
3) _se_die "no free stack index (0..15 all taken); free one with 'mise run local:stack-release' in an unused checkout, or 'mise run local:stack-prune' to reap deleted checkouts" ;;mise/tasks/local/stack-prune (sketch)
#!/usr/bin/env bash
set -euo pipefail
#MISE description="Reap registry slots of checkouts that were deleted without stack-release"
FLOW_LOCAL="${HOME}/flow-local"
REGISTRY="${FLOW_LOCAL}/stacks.tsv"
LOCKFILE="${FLOW_LOCAL}/stacks.lock"
[ -f "${REGISTRY}" ] || { echo "No registry present; nothing to prune."; exit 0; }
(
flock 9
tmp="$(mktemp "${FLOW_LOCAL}/stacks.tsv.XXXXXX")"
while IFS=$'\t' read -r idx name root; do
[ -n "${root}" ] || continue
if [ -d "${root}" ]; then # live checkout: keep
printf '%s\t%s\t%s\n' "${idx}" "${name}" "${root}" >> "${tmp}"; continue
fi
cluster="local-${name}-cluster"
if systemctl --user list-units --no-legend --state=active \
"flow-*@${name}.service" "flow-*@${name}.target" \
"flow-*@${cluster}.service" "flow-*@${cluster}.target" \
"flow-*@${cluster}-*.service" "flow-*@${cluster}-*.target" 2>/dev/null \
| grep -q .; then
echo "prune: KEEPING '${name}' (idx ${idx}): ${root} is gone but its units still run." >&2
echo "prune: stop first: systemctl --user stop 'flow-*@${name}.*' 'flow-*@${cluster}*'" >&2
printf '%s\t%s\t%s\n' "${idx}" "${name}" "${root}" >> "${tmp}"; continue
fi
target="${HOME}/cargo-target/${name}"
[ -n "${name}" ] && [ "${target}" != "${HOME}/cargo-target" ] && [ -d "${target}" ] && rm -rf "${target}"
statedir="${FLOW_LOCAL}/${name}"
[ -n "${name}" ] && [ "${statedir}" != "${FLOW_LOCAL}" ] && [ -d "${statedir}" ] && rm -rf "${statedir}"
if command -v docker >/dev/null 2>&1; then
while IFS= read -r vol; do
[ "${vol##*_}" = "${name}" ] || continue
docker volume rm "${vol}" >/dev/null 2>&1 || true
done < <(docker volume ls --format '{{.Name}}' 2>/dev/null | grep '^supabase_' || true)
fi
echo "prune: reaped '${name}' (idx ${idx}); checkout ${root} is gone."
done < "${REGISTRY}"
mv "${tmp}" "${REGISTRY}"
) 9>"${LOCKFILE}"Non-blocking: the workflow works today; this is about the failure mode after checkouts accumulate and get deleted casually.
There was a problem hiding this comment.
I re-worked stack release to be lazy. Deleting a worktree is the functional release mechanism, and then an ambient mise task (which can also be run explicitly) detects the deletion and removes everything else.
|
PTAL |
Lay the foundation for isolated per-checkout stacks: every checkout (primary clone or linked git worktree) gets its own stack identity, and mise becomes the mandatory entry point that makes that identity ambient. - stack-env allocates a per-checkout identity — name = sanitized basename, index = lowest free slot in 0..15 — in a flock-guarded, machine-global registry (stacks.tsv). It derives this stack's ports (base(i) = 10000 + 1000*i; the last three digits identify the service uniformly) and exports FLOW_* / CARGO_TARGET_DIR / GOBIN / DATABASE_URL ambiently via mise `[[env]] _.source`, putting built binaries on PATH via a serial `_.path` directive. There is no canonical stack. - lib.sh holds the shared shell helpers; the old `common` task is removed. - stack / stop / stack-release manage the lifecycle: stop scopes teardown to one stack, and stack-release frees the registry slot and reclaims the stack's footprint — build artifacts (tens of GB under CARGO_TARGET_DIR), the Supabase Docker volumes `supabase stop` preserves, and the private state dir — each rm guarded so a short value can't escalate to a shared root. stack-info surfaces this checkout's ports, units, and commands. - DATABASE_URL is exported per-stack (sqlx's compile-time query! macros connect to it), so .cargo/config.toml drops its static localhost:5432 default, which no longer matches any stack. - Cargo.toml disables incremental compilation: sccache won't cache incremental compiles (workspace crates would bypass the shared cache), and the incremental/ dir dominates each stack's target-dir footprint while sharing nothing across checkouts.
With each checkout owning a stack identity and ambient environment, wire the actual services to that identity so multiple stacks coexist. - systemd user units become templates (flow-*@.service) instanced on the stack or data-plane name. The per-service mise tasks generate each instance's env file and drop-ins, wiring cross-unit dependencies and binding every service to this stack's ports. - supabase/config.toml resolves project id, ports, and site URL through env() substitution, failing loud when a variable is unset, so the Supabase instance lands on this stack's ports too. - build tasks and vm/port-forward, and the local helper scripts and ops-catalog fixtures, follow the same per-stack ports and endpoints.
Client and CLI code can no longer assume fixed local ports now that each stack binds its own. Have them read this checkout's endpoints from the ambient environment instead of compiled-in defaults. - flowctl drops its compiled-in `local` profile defaults; each stack's profile is generated and its URLs refresh on every start, while login tokens are preserved. - dekaf takes its API / AGENT endpoints from its generated env file. - flow-client / flow-client-next and the flowctl-go temp-data-plane command resolve local endpoints from the same per-stack variables.
Adapt tests and CI to per-checkout stacks: everything now reads FLOW_PG_URL / FLOW_CLUSTER / FLOW_PORT_* and this stack's build dirs, and fails loudly when run outside a mise stack. - Rust and Go test harnesses (agent, automations, catalog-stats, dekaf e2e) source their endpoints from the ambient stack env. - e2e-support resolves the gazette broker from GOBIN when set — the mise path builds it into this stack's GOBIN, not the shared ~/go/bin — falling back to ~/go/bin outside a mise stack. - CI workflows and the ci/gotest, ci/catalog-test, ci/dekaf-e2e tasks use the ambient CARGO_TARGET_DIR / GOBIN so they find the Rust and Go binaries this stack actually builds (including flow-connector-init in the musl target dir), and cache per-stack target dirs. - bootstrap/ide-settings points editor tooling at the per-checkout dirs.
- local/README.md is rewritten around per-checkout stacks: mise as the mandatory entry point, the base(i) = 10000 + 1000*i port scheme, no canonical stack, and scoped teardown vs. slot release. - AGENTS.md records the per-stack mise workflow for future agents. - tests/soak/README.md follows the new ports and commands.
Per-checkout target dirs are 20-60G each and overlap heavily across checkouts. Back ~/cargo-target on dev VMs with a sparse loopback btrfs image (50% of disk, capping physical post-compression bytes): - zstd:3 transparent compression (~3.2x measured on real artifacts) - a 10-minute duperemove + fstrim timer collapses shared extents and punches freed space back into the sparse image (~26% of logical bytes deduped even across divergent branches; near-total for same-commit worktrees, extent-level dedupe shares blocks of differing files too) - loop device uses direct-io, via a systemd oneshot unit rather than fstab, which cannot express it bootstrap:cargo-target-fs is idempotent, Linux-only, and never migrates an existing plain-dir ~/cargo-target (VMs are disposable; recreate to adopt). vm:create-post invokes it before anything builds. GOCACHE moves under ~/cargo-target so the large Go build cache lands on the filesystem; Go stores it uncompressed, so btrfs's single pass does real work on it. SCCACHE_DIR is deliberately left at its default: sccache already compresses its own entries, so a compressed blob would only be recompressed for ~no gain under the compress-force mount (which can't be told to skip an already-compressed subtree). CI is unaffected: it uses sccache's GHA backend, and no workflow caches the Go path.
The single signal that frees a stack's slot and artifacts is now that its registered checkout directory is gone. Being *stopped* releases nothing: a stopped-but-present checkout keeps its index and its (tens-of-GB) build cache, so restarting is instant. You release a stack by deleting its worktree; a garbage collector then reclaims it in the background. Replaces the explicit `local:stack-release` task with: - reap-lib.sh: shared reaper sourced by stack-env, stack, stop, and the new stack-prune task. reap_teardown_stack tears a stack down by name alone (its checkout is gone, so nothing is ambient) — stopping orphaned units first so a reused index can't collide on ports, then reclaiming units/drop-ins/env files, the build target, the state dir, and Supabase volumes/network. Guarded against empty/short names and prefix over-match; volume matching is exact-suffix so 'flow' can't reap 'flow-2'. reap_gone_entries rewrites the registry to only the entries whose root still exists. - stack-prune: idempotent task that runs the reaper under the registry lock. - Auto-reap at natural joints: local:stack and local:stop call the reaper, and stack-env reaps on the (rare) new-checkout allocation path — so a registry full of deleted checkouts heals itself instead of dead-ending on "no free index". Also guard `dekaf --local` with no endpoint: local ports are dynamic so there is no local default, and clap would otherwise fall through to the *production* defaults, pointing a local anon token at prod. Fail loud instead.
38b5847 to
767928d
Compare
## What's Changed * runtime: periodically store primary FSMHints of V2 shards by @williamhbaker in estuary/flow#3027 * mise build:flowctl for a local flowctl binary by @mdibaiee in estuary/flow#3031 * go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3033 * data-plane-controller: dns and s3 passthroughs by @qaoj in estuary/flow#3022 * Self-service private-link configuration by @jshearer in estuary/flow#2944 * supabase: scoped refresh tokens so CI writes via PostgREST instead of direct psql by @skord in estuary/flow#3013 * docs: Describe SQL Server replica suport by @willdonnelly in estuary/flow#3040 * proto-gazette: regenerate to pick up SUSPEND_KEEP by @williamhbaker in estuary/flow#3041 * docs: correct custom-column-types backfill behavior for DDL changes by @jwhartley in estuary/flow#3036 * Refresh token GraphQL operations and token exchange endpoint by @GregorShear in estuary/flow#3020 * docs: shopify stream & resource config additions by @Alex-Bair in estuary/flow#3034 * validation/collection: exclude `flow://inferred-schema` from the managed-defs redact check by @jshearer in estuary/flow#3046 * docs: update Bigtable connector permissions example by @mwillman-estuary in estuary/flow#3045 * docs: disable auto-discover for multi-binding file source captures by @jwhartley in estuary/flow#3037 * docs: add streams and API pinning details to stripe-native by @nicolaslazo in estuary/flow#3055 * data-plane-controller: restart setting by @qaoj in estuary/flow#3048 * deploy: route control-plane services through the VPC NAT for a stable egress IP by @skord in estuary/flow#3044 * docs: document the Exclude Flow Document (no_flow_document) materialization option by @jwhartley in estuary/flow#3018 * oidc-discovery-server: survive direct VPC egress cold-start on deploy by @skord in estuary/flow#3062 * docs: clarify retain_existing_data_on_backfill requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3049 * data-plane-controller: restart on ansiblehost by @qaoj in estuary/flow#3066 * flow-web: Bump version to release by @jshearer in estuary/flow#3068 * ops-catalog: misc updates by @williamhbaker in estuary/flow#3067 * proto-flow: tolerate unknown JSON fields in connector protocols by @williamhbaker in estuary/flow#3059 * flowctl: add raw split-shards to scale out V2 tasks by @williamhbaker in estuary/flow#3021 * supabase: qualify replace_data_plane_releases DELETE for PostgREST by @skord in estuary/flow#3076 * docs: kcat recipes for testing a Dekaf topic by @jwhartley in estuary/flow#3072 * Docs: document per-prefix alert scoping and configurable thresholds by @jwhartley in estuary/flow#3039 * notifications: trial bucket retention is 20 days, not 30 by @jwhartley in estuary/flow#3074 * runtime/container: use ops::decode::Decoder for log lines by @williamhbaker in estuary/flow#3091 * data-plane-controller: remove restart flag by @qaoj in estuary/flow#3087 * runtime-next: durably seed initial connector state as {} to match V1 by @williamhbaker in estuary/flow#3081 * runtime-next: don't abandon the final capture transaction on connector EOF by @jgraettinger in estuary/flow#3090 * Docs: Remove an IP address from GCP allowed list by @jwhartley in estuary/flow#3080 * docs: remove static data plane IP list, reference dashboard by @jwhartley in estuary/flow#3097 * dekaf: improve logging by @williamhbaker in estuary/flow#3078 * agent: make startup logging consistent by @williamhbaker in estuary/flow#3098 * shuffle: make the shuffle disk limit configurable per-task by @jgraettinger in estuary/flow#3096 * runtime-next: automatically split journals under sustained append-rate throttling by @dgreer-dev in estuary/flow#3029 * billing: add tenant billing contact fields and per-tenant controller by @jshearer in estuary/flow#2902 * flowctl-go(api test): raise test-shard readiness window from ~3s to ~30s by @jshearer in estuary/flow#3101 * runtime: strip the V2 committed-close marker for derivations by @williamhbaker in estuary/flow#3100 * deps: bump aws-lc-sys and lz4_flex for security advisories by @skord in estuary/flow#2875 * data-plane-controller: fix db timeout by @qaoj in estuary/flow#3105 * agent-api: survive direct VPC egress cold-start on instance startup by @skord in estuary/flow#3109 * docs: avoiding backfills during a database failover or host change by @jwhartley in estuary/flow#3086 * docs: clarify column-level SELECT grants unsupported for MySQL/MariaDB CDC by @jwhartley in estuary/flow#3085 * json: require RFC3339 'T' separator in date-time format validator by @jacobmarble in estuary/flow#3116 * control-plane: reserve privileged tenant names by @jwhartley in estuary/flow#3083 * agent-api: harden startup and shutdown on Cloud Run by @skord in estuary/flow#3114 * Incidental fixes: gazette Append retry, connector-init image, materialize tear-down by @jgraettinger in estuary/flow#3121 * agent: flag to create new captures as runtime-v2 by @williamhbaker in estuary/flow#3107 * runtime-next: add minimum interval for post-txn triggers by @dgreer-dev in estuary/flow#3110 * flowctl: chunk API requests by url size rather than fixed count by @mdibaiee in estuary/flow#3123 * dekaf: use projection_constraints only, no constraints by @mdibaiee in estuary/flow#3124 * flowctl: normalize --prefix trailing slash to avoid misleading PermissionDenied by @GregorShear in estuary/flow#3075 * docs: retain_existing_data_on_backfill no longer requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3125 * Restart Materializations sessions before IAM token expiry by @dgreer-dev in estuary/flow#3130 * docs: add "bring your own app" instructions for source-quickbooks by @nicolaslazo in estuary/flow#3135 * flowctl: re-tool `preview` on the runtime-next stack by @jgraettinger in estuary/flow#3117 * Adding support for updating the quotas when payment info changes by @bbartman in estuary/flow#3127 * control-plane-api: add unauthenticated publicDataPlanes GraphQL query by @GregorShear in estuary/flow#3129 * go/bindings: update snapshots for materialize-sqlite ser_policy by @dgreer-dev in estuary/flow#3142 * Revert "flowctl: replace `preview` with the runtime-next implementation" by @jgraettinger in estuary/flow#3141 * local: per-checkout stacks — every checkout runs its own isolated stack by @jgraettinger in estuary/flow#3137 * dekaf: log tls handshake eof instead of returning as error by @danielnelson in estuary/flow#3143 * Updating migrations to better support testing. by @bbartman in estuary/flow#3146 * shuffle: raise causal-hint stall timeout to 15m; lower prune horizon to 2GB by @jgraettinger in estuary/flow#3148 * ci: fix Platform Build by packaging Go binaries from per-checkout $GOBIN by @jgraettinger in estuary/flow#3149 * capture+materialize: Support Nonsensitive Field Overlays by @willdonnelly in estuary/flow#3119 * Docs: The Great Description-ing by @aeluce in estuary/flow#3057 * runtime: fix recursive read-lock deadlock in capture-v2 buildJoin by @jgraettinger in estuary/flow#3156 * validation: surface write-schema redact annotations on projections of split-schema collections by @GregorShear in estuary/flow#3147 * control-plane-api: retry Stripe customer search on index-lag misses by @jgraettinger in estuary/flow#3157 * flowctl: multi-shard --fixture for raw preview-next by @mdibaiee in estuary/flow#3154 * dekaf: prevent cached `TimeoutNoData` fetch responses from incorrectly signaling EOF by @jshearer in estuary/flow#3153 * mise: remove Lima VM host share and other VM tweaks by @jgraettinger in estuary/flow#3163 * docs: document connecting Azure Private Link to native Azure resources by @jwhartley in estuary/flow#3138 * docs: explain why exclusiveCollectionFilter needs few enabled bindings by @jwhartley in estuary/flow#3126 ## New Contributors * @bbartman made their first contribution in estuary/flow#3127 **Full Changelog**: estuary/flow@v0.6.10...v0.6.11 Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>
Why
Until now the local platform stack was effectively a singleton: one database, one
set of ports, one build directory, one flowctl profile per machine. That's fine
when you're the only thing running, but it makes parallel work painful. Spin up a
second checkout to QA a fix while a soak test runs in the first, and the two
stacks fight over the same ports and the same Postgres. In practice you had to
tear one down before bringing the other up — and with multiple people (or agents)
sharing a dev VM, you were constantly stepping on each other.
This change makes a local stack belong to a checkout rather than to the machine.
Every clone and every git worktree gets its own fully isolated platform — its own
database, etcd, brokers, reactors, ports, build artifacts, and flowctl profile —
so any number of them can run side by side without coordination.
What it feels like to use
The whole point is that you don't think about isolation. You work in a checkout,
and everything scopes to it automatically:
stack-infois the one thing to remember: it prints the ports and commands forwherever you happen to be, so you never compute a port or pick a name. Under the
hood each checkout claims a small integer index and derives everything from it,
but as a developer you mostly just see that things don't collide.
A few niceties fall out of this:
flowctljust works. Inside a checkout, bareflowctltargets thatcheckout's stack — no
--profileflag to remember, and no chance ofaccidentally driving the wrong stack.
FLOW_PG_URL,FLOW_CLUSTER, theFLOW_PORT_*set, and friends are already in your environment when you runthrough mise, so
psql "$FLOW_PG_URL"and the like do the right thing.directory, so rebuilds in one worktree don't invalidate another's cache.
local:stoptears down just the stack you're in andleaves everyone else running.
Lifecycle: stop is cheap, deleting the worktree is the release
Stopping and releasing are deliberately different actions:
local:stopis a stack-scoped guillotine — it stops this stack's units andwipes its runtime state, but keeps its registry slot, its build cache, and
its Supabase volumes, so a later start is instant.
Releasing a slot happens when you delete the checkout. A deleted checkout
directory is the one and only signal that frees a slot and its artifacts.
Just remove the worktree when you're done:
git worktree remove ../feature-x # (or plain rm -rf) — that's the releaseA garbage collector reclaims deleted checkouts in the background — stopping their
orphaned units, then freeing the slot, build target, state dir, and Supabase
volumes. It runs automatically at natural joints (
local:stackandlocal:stop), so starting or stopping a stack anywhere cleans up worktrees you'veremoved. You can also run it on demand:
mise run local:stack-prune # reclaim slots of deleted checkouts (idempotent)This also means a registry full of removed checkouts heals itself instead of
dead-ending on "no free index": the new-checkout allocation path reaps gone
entries before it looks for a free slot.
A note on mise
Because stack identity now comes from the checkout, mise is the mandatory entry
point — it's what makes the per-checkout variables ambient. Running tasks and
tests through mise (
mise run ...,mise exec -- ...) always has the rightenvironment. Outside mise, wrap once with
eval "$(mise run local:stack-env)".Scripts and tests fail loudly rather than silently defaulting if the environment
is missing, so you find out immediately instead of debugging a mystery stack.
In the same spirit,
dekaf --localnow fails loud when no endpoint is given:local-stack ports are dynamic so there is no compiled local default, and clap
would otherwise fall through to the production defaults — pointing a local anon
token at prod.
mise run local:data-planesupplies the right endpoints; run itthat way, or pass
API_ENDPOINT/AGENT_ENDPOINTexplicitly.local/README.mdis the full reference for how this is wired together.Keeping the disk in check on dev VMs
Per-checkout isolation has a cost: every checkout gets its own
~/cargo-target/<name>/, each 20–60G, and they overlap heavily — the same cratescompiled in worktree after worktree. On a shared dev VM with several checkouts
that adds up fast.
So on dev VMs,
~/cargo-targetis now backed by a compressed,self-deduplicating btrfs filesystem on a sparse loopback image sized to 50% of
the disk:
duperemove+fstrimtimer that collapses extents sharedacross checkouts and punches the freed space back into the sparse image —
near-total dedupe for same-commit worktrees, and ~26% of logical bytes even
across divergent branches (extent-level dedupe shares blocks of differing files
too). Net: many divergent checkouts fit in a fraction of their raw footprint.
and it compresses well. sccache stays at its default
~/.cache/sccache: italready compresses its own entries, so it would gain nothing under the
compress-forcemount (and can't be excluded from it).It's set up by
bootstrap:cargo-target-fs, invoked fromvm:create-postbeforeanything builds. The task is idempotent and Linux-only, and it never migrates an
existing plain-dir
~/cargo-target(VMs are disposable, make a new one).QA
Exercised end to end: full bring-up, tenant provisioning, a soak test with all
verification checks passing, isolation spot-checks across concurrent stacks, and
scoped teardown.
The slot-reclaim lifecycle was driven directly against a live stack:
(
alphadoesn't touchalpha-2), exact-suffix volume matching (flowdoesn'treap
flow-2), and registry rewrite (including the all-gone empty case) allhold.
build target, state dir, and a Supabase volume, deleted the worktree, and
confirmed
local:stack-prunereclaimed exactly that entry (slot, target, state,volume) while a concurrently-running stack was left completely untouched. The
freed index was then immediately reused by a new checkout.
local:stackandlocal:stopreclaimdeleted checkouts, and that
local:stopretains the current stack's slot, buildcache, and Supabase volumes for an instant restart.
--localguard — verified it bails with an actionable message whenAPI_ENDPOINT/AGENT_ENDPOINTare missing, and starts cleanly on thelocal:data-planehappy path.