runtime: persist the max-key fail-safe sentinel at adoption - #3191
#3191Merged
jgraettinger merged 1 commit intoJul 17, 2026
masterestuary/flow:masterfrom
johnny/runtime-max-key-persistence-9de575estuary/flow:johnny/runtime-max-key-persistence-9de575Copy head branch name to clipboard
Merged
runtime: persist the max-key fail-safe sentinel at adoption#3191jgraettinger merged 1 commit intomasterestuary/flow:masterfrom johnny/runtime-max-key-persistence-9de575estuary/flow:johnny/runtime-max-key-persistence-9de575Copy head branch name to clipboard
jgraettinger merged 1 commit into
masterestuary/flow:masterfrom
johnny/runtime-max-key-persistence-9de575estuary/flow:johnny/runtime-max-key-persistence-9de575Copy head branch name to clipboard
Conversation
The V1 runtime can commit checkpoint progress for a binding without ever observing one of its keys (e.g. via ACK-only journal reads), in which case no MK-v2: max-key is persisted. On the next recovery the binding is active in the checkpoint without a persisted max-key, and recovery adopts a [0xff] fail-safe sentinel in memory only. Because no packed key compares greater than [0xff], the binding's max-key never ratchets again and is never persisted -- permanently. This is end-to-end safe within V1 (the sentinel conservatively disables the load-skip optimization), but it breaks V1 => V2 migration semantics: the V2 runtime interprets a missing max-key as "no keys were ever stored", which enables the load-skip optimization for a binding that may hold extensive stored data. Close the loop by durably persisting the sentinel at the moment of adoption, synchronized to the recovery log before any Load or Store of the session. A missing MK-v2: key then once again means "no keys ever stored", matching V2's native invariant, and migration requires no V2-side changes. Recovery must also accept the persisted sentinel: it is intentionally not a valid packed tuple (0xff is tuple::ESCAPE, above every type code), so it is matched exactly and used verbatim rather than unpacked. Any other unparseable value still fails as corruption. This same relaxation makes a V2 => V1 rollback of a task carrying the sentinel recover cleanly.
Member
Author
|
Sanity checked on a local stack. |
mdibaiee
added a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Jul 23, 2026
## What's Changed * docs: surface MCP and agent skills for coding agents by @jwhartley in estuary/flow#3093 * dekaf: serve the document containing a mid-document fetch offset by @jshearer in estuary/flow#3150 * runtime: let docker atomically assign published connector host ports by @dgreer-dev in estuary/flow#3162 * runtime-next: discard hinted frontier when committed frontier is rebuilt by @jgraettinger in estuary/flow#3165 * docs/source-postgres: Sync statement_timeout change by @willdonnelly in estuary/flow#3168 * Provide capture/materializations created_at date by @dgreer-dev in estuary/flow#3160 * docs: add Processing order section for materialization binding priority by @jwhartley in estuary/flow#3171 * docs: reorganize agent skills page by plugin, add derivations and schema by @jwhartley in estuary/flow#3170 * go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3180 * Service Accounts by @GregorShear in estuary/flow#3058 * Bound too-large error messages over RPC by @dgreer-dev in estuary/flow#3182 * control-plane: guarded, auto-expiring temporary support access to restricted tenants by @skord in estuary/flow#3115 * private links: model links as rows with controller-observed status by @jshearer in estuary/flow#3063 * Adding support for an stripe web hook endpoint. by @bbartman in estuary/flow#3174 * docs: Dekaf consumer behaviors + Spark Structured Streaming guidance by @jwhartley in estuary/flow#3094 * shuffle: log canonical journal name in slice read events by @jgraettinger in estuary/flow#3190 * runtime: persist the max-key fail-safe sentinel at adoption by @jgraettinger in estuary/flow#3191 * local: make flow-plane-link robust to control-plane agent start-up by @jgraettinger in estuary/flow#3158 * gazette: surface fragment store diagnostic on FRAGMENT_STORE_UNHEALTHY by @jgraettinger in estuary/flow#3159 * docs: add warning about source-stripe-native's events API version by @nicolaslazo in estuary/flow#3166 * agent-api: Add `STRIPE_WEBHOOK_SECRET` to cloud run config by @jshearer in estuary/flow#3196 * shuffle, gazette: guard against wedged reads by @jgraettinger in estuary/flow#3193 * runtime-next: gate task-log stream by configured log level by @williamhbaker in estuary/flow#3198 * flowctl: preview-next appends a final drain session by @mdibaiee in estuary/flow#3195 * Allow localhost:3000 origin for the local control-plane agent by @GregorShear in estuary/flow#3206 * control-plane-api: don't lock injected ops collections during publications by @skord in estuary/flow#3164 * runtime-next: force txn close at any usage ceiling, bypassing min duration by @jgraettinger in estuary/flow#3203 * shuffle: replay gapped producers on restart by @jgraettinger in estuary/flow#3202 * docs: source-zuora by @Alex-Bair in estuary/flow#3204 * dekaf: only read back when a fetch offset lands inside a document by @mdibaiee in estuary/flow#3205 * mise: dev-VM zone configurability (create fallback, zone-agnostic SSH, vm:move-gcp) by @skord in estuary/flow#3208 * docs/postgres: Discovery Filters by @willdonnelly in estuary/flow#3215 * docs: document materialize-s3-iceberg nanosecond_timestamps option by @jacobmarble in estuary/flow#3199 * runtime v2 flag for new captures and derivations by @williamhbaker in estuary/flow#3219 * runtime-next: converge startup reconciliation via rescan Persists by @jgraettinger in estuary/flow#3218 * build(deps): bump fast-uri from 3.1.0 to 3.1.4 in /site by @dependabot[bot] in estuary/flow#3223 * build(deps): bump svgo from 3.3.3 to 3.3.4 in /site by @dependabot[bot] in estuary/flow#3224 * build(deps): bump dompurify from 3.3.3 to 3.4.12 in /site by @dependabot[bot] in estuary/flow#3222 * build(deps): bump shell-quote from 1.8.3 to 1.10.0 in /site by @dependabot[bot] in estuary/flow#3213 * build(deps): bump body-parser from 1.20.4 to 1.20.6 in /site by @dependabot[bot] in estuary/flow#3212 * build(deps): bump webpack-dev-server from 5.2.2 to 5.2.6 in /site by @dependabot[bot] in estuary/flow#3211 * build(deps): bump websocket-driver from 0.7.4 to 0.7.5 in /site by @dependabot[bot] in estuary/flow#3183 * Docs: MySQL discovery filters by @aeluce in estuary/flow#3217 * docs: source-smartsheet by @nicolaslazo in estuary/flow#3228 * control-plane-api: surface per-store health diagnostics in storage-mapping mutations by @GregorShear in estuary/flow#3181 * async-process: handle reaping-task JoinError in Child::wait() without panicking by @jgraettinger in estuary/flow#3225 * feat: end-to-end truncation by @williamhbaker in estuary/flow#3132 * docs: source-zuora uses the AQuA API now by @Alex-Bair in estuary/flow#3237 * Improve storage mapping GraphQL results by @GregorShear in estuary/flow#3200 * control-plane: add `closed` flag to data planes by @GregorShear in estuary/flow#3175 **Full Changelog**: estuary/flow@v0.6.11...v0.6.12 Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>
mdibaiee
added a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Jul 23, 2026
## What's Changed * docs: surface MCP and agent skills for coding agents by @jwhartley in estuary/flow#3093 * dekaf: serve the document containing a mid-document fetch offset by @jshearer in estuary/flow#3150 * runtime: let docker atomically assign published connector host ports by @dgreer-dev in estuary/flow#3162 * runtime-next: discard hinted frontier when committed frontier is rebuilt by @jgraettinger in estuary/flow#3165 * docs/source-postgres: Sync statement_timeout change by @willdonnelly in estuary/flow#3168 * Provide capture/materializations created_at date by @dgreer-dev in estuary/flow#3160 * docs: add Processing order section for materialization binding priority by @jwhartley in estuary/flow#3171 * docs: reorganize agent skills page by plugin, add derivations and schema by @jwhartley in estuary/flow#3170 * go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3180 * Service Accounts by @GregorShear in estuary/flow#3058 * Bound too-large error messages over RPC by @dgreer-dev in estuary/flow#3182 * control-plane: guarded, auto-expiring temporary support access to restricted tenants by @skord in estuary/flow#3115 * private links: model links as rows with controller-observed status by @jshearer in estuary/flow#3063 * Adding support for an stripe web hook endpoint. by @bbartman in estuary/flow#3174 * docs: Dekaf consumer behaviors + Spark Structured Streaming guidance by @jwhartley in estuary/flow#3094 * shuffle: log canonical journal name in slice read events by @jgraettinger in estuary/flow#3190 * runtime: persist the max-key fail-safe sentinel at adoption by @jgraettinger in estuary/flow#3191 * local: make flow-plane-link robust to control-plane agent start-up by @jgraettinger in estuary/flow#3158 * gazette: surface fragment store diagnostic on FRAGMENT_STORE_UNHEALTHY by @jgraettinger in estuary/flow#3159 * docs: add warning about source-stripe-native's events API version by @nicolaslazo in estuary/flow#3166 * agent-api: Add `STRIPE_WEBHOOK_SECRET` to cloud run config by @jshearer in estuary/flow#3196 * shuffle, gazette: guard against wedged reads by @jgraettinger in estuary/flow#3193 * runtime-next: gate task-log stream by configured log level by @williamhbaker in estuary/flow#3198 * flowctl: preview-next appends a final drain session by @mdibaiee in estuary/flow#3195 * Allow localhost:3000 origin for the local control-plane agent by @GregorShear in estuary/flow#3206 * control-plane-api: don't lock injected ops collections during publications by @skord in estuary/flow#3164 * runtime-next: force txn close at any usage ceiling, bypassing min duration by @jgraettinger in estuary/flow#3203 * shuffle: replay gapped producers on restart by @jgraettinger in estuary/flow#3202 * docs: source-zuora by @Alex-Bair in estuary/flow#3204 * dekaf: only read back when a fetch offset lands inside a document by @mdibaiee in estuary/flow#3205 * mise: dev-VM zone configurability (create fallback, zone-agnostic SSH, vm:move-gcp) by @skord in estuary/flow#3208 * docs/postgres: Discovery Filters by @willdonnelly in estuary/flow#3215 * docs: document materialize-s3-iceberg nanosecond_timestamps option by @jacobmarble in estuary/flow#3199 * runtime v2 flag for new captures and derivations by @williamhbaker in estuary/flow#3219 * runtime-next: converge startup reconciliation via rescan Persists by @jgraettinger in estuary/flow#3218 * build(deps): bump fast-uri from 3.1.0 to 3.1.4 in /site by @dependabot[bot] in estuary/flow#3223 * build(deps): bump svgo from 3.3.3 to 3.3.4 in /site by @dependabot[bot] in estuary/flow#3224 * build(deps): bump dompurify from 3.3.3 to 3.4.12 in /site by @dependabot[bot] in estuary/flow#3222 * build(deps): bump shell-quote from 1.8.3 to 1.10.0 in /site by @dependabot[bot] in estuary/flow#3213 * build(deps): bump body-parser from 1.20.4 to 1.20.6 in /site by @dependabot[bot] in estuary/flow#3212 * build(deps): bump webpack-dev-server from 5.2.2 to 5.2.6 in /site by @dependabot[bot] in estuary/flow#3211 * build(deps): bump websocket-driver from 0.7.4 to 0.7.5 in /site by @dependabot[bot] in estuary/flow#3183 * Docs: MySQL discovery filters by @aeluce in estuary/flow#3217 * docs: source-smartsheet by @nicolaslazo in estuary/flow#3228 * control-plane-api: surface per-store health diagnostics in storage-mapping mutations by @GregorShear in estuary/flow#3181 * async-process: handle reaping-task JoinError in Child::wait() without panicking by @jgraettinger in estuary/flow#3225 * feat: end-to-end truncation by @williamhbaker in estuary/flow#3132 * docs: source-zuora uses the AQuA API now by @Alex-Bair in estuary/flow#3237 * Improve storage mapping GraphQL results by @GregorShear in estuary/flow#3200 * control-plane: add `closed` flag to data planes by @GregorShear in estuary/flow#3175 **Full Changelog**: estuary/flow@v0.6.11...v0.6.12 Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Persist the max-key fail-safe sentinel at the moment of adoption, so the persisted state once again matches V2's native invariant that a missing
MK-v2:key means "no keys were ever stored."Background
The V1 runtime can commit checkpoint progress for a binding without ever observing one of its keys (e.g. via ACK-only journal reads), in which case no
MK-v2:max-key is persisted. On the next recovery the binding is active in the checkpoint without a persisted max-key, and recovery adopts a[0xff]fail-safe sentinel in memory only. Because no packed key compares greater than[0xff], the binding's max-key never ratchets again and is never persisted — permanently.This is end-to-end safe within V1 (the sentinel conservatively disables the load-skip optimization), but it breaks V1 → V2 migration semantics: the V2 runtime interprets a missing max-key as "no keys were ever stored", which enables the load-skip optimization for a binding that may hold extensive stored data.
Change
MK-v2:key then once again means "no keys ever stored", matching V2's native invariant, so migration requires no V2-side changes.0xffistuple::ESCAPE, above every type code), so it is matched exactly and used verbatim rather than unpacked. Any other unparseable value still fails as corruption. This same relaxation makes a V2 → V1 rollback of a task carrying the sentinel recover cleanly.