Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

validation/collection: exclude flow://inferred-schema from the managed-defs redact check - #3046

#3046
Merged
jshearer merged 1 commit into
masterestuary/flow:masterfrom
jshearer/fix_redact_inferred_schema_false_positiveestuary/flow:jshearer/fix_redact_inferred_schema_false_positiveCopy head branch name to clipboard
Jun 17, 2026
Merged

validation/collection: exclude flow://inferred-schema from the managed-defs redact check#3046
jshearer merged 1 commit into
masterestuary/flow:masterfrom
jshearer/fix_redact_inferred_schema_false_positiveestuary/flow:jshearer/fix_redact_inferred_schema_false_positiveCopy head branch name to clipboard

Conversation

@jshearer

Copy link
Copy Markdown
Contributor

#2931 added RedactInsideManagedDefs, which scans a collection's $defs for redact annotations inside system-managed schema subtrees (flow://connector-schema, flow://write-schema, flow://relaxed-write-schema, flow://inferred-schema) and rejects the publication, on the basis that those subtrees are overwritten on every build so a redact placed inside one is silently lost. Including flow://inferred-schema in that set was incorrect.

The regression

When a collection uses schema inference, its redact annotations flow from the write schema into the inferred schema, and validation inlines that inferred schema into $defs/flow://inferred-schema before it runs the footgun check. The check found the redact there and rejected the publication, telling the user to "move the annotation to the top level" when it already was at the top level of the write schema.

Because the controller only republishes a collection when its inferred schema changes, the failure surfaced per-collection whenever that collection's inferred schema next updated, at which point its background publication entered a failed-publish loop. The check also couldn't ever flag a genuine misplacement inside flow://inferred-schema: that subtree is overwritten with the inferred schema on every build before the check ran.

The fix

Drop flow://inferred-schema from the scanned IDs. A redact inside it reflects the write schema rather than a misplacement, and the dangerous case (a redact in the effective read schema with no write-schema counterpart) is still caught by ReadSchemaRedactNotInWriteSchema. The other three managed subtrees remain scanned

…ged-defs redact check

When a collection uses schema inference, its `redact` annotations flow from the write schema into the inferred schema, which validation inlines into `$defs/flow://inferred-schema` before checking it. The placement check treated those annotations as a misplacement and rejected the publication, so any collection that pairs schema inference with redaction entered a failed-publish loop once its inferred schema next updated.

Drop `flow://inferred-schema` from the scanned IDs: a `redact` found there reflects the correctly-placed write-schema annotation rather than a user misplacement, and the subtree is overwritten on every build regardless. `flow://write-schema`, `flow://relaxed-write-schema`, and `flow://connector-schema` are still scanned.
@jshearer
jshearer requested a review from a team June 17, 2026 18:02
@jshearer
jshearer merged commit cd0a34a into master Jun 17, 2026
11 checks passed
mdibaiee added a commit to estuary/homebrew-flowctl that referenced this pull request Jul 15, 2026
## What's Changed
* runtime: periodically store primary FSMHints of V2 shards by @williamhbaker in estuary/flow#3027
* mise build:flowctl for a local flowctl binary by @mdibaiee in estuary/flow#3031
* go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3033
* data-plane-controller: dns and s3 passthroughs by @qaoj in estuary/flow#3022
* Self-service private-link configuration by @jshearer in estuary/flow#2944
* supabase: scoped refresh tokens so CI writes via PostgREST instead of direct psql by @skord in estuary/flow#3013
* docs: Describe SQL Server replica suport by @willdonnelly in estuary/flow#3040
* proto-gazette: regenerate to pick up SUSPEND_KEEP by @williamhbaker in estuary/flow#3041
* docs: correct custom-column-types backfill behavior for DDL changes by @jwhartley in estuary/flow#3036
* Refresh token GraphQL operations and token exchange endpoint by @GregorShear in estuary/flow#3020
* docs: shopify stream & resource config additions by @Alex-Bair in estuary/flow#3034
* validation/collection: exclude `flow://inferred-schema` from the managed-defs redact check by @jshearer in estuary/flow#3046
* docs: update Bigtable connector permissions example by @mwillman-estuary in estuary/flow#3045
* docs: disable auto-discover for multi-binding file source captures by @jwhartley in estuary/flow#3037
* docs: add streams and API pinning details to stripe-native by @nicolaslazo in estuary/flow#3055
* data-plane-controller: restart setting by @qaoj in estuary/flow#3048
* deploy: route control-plane services through the VPC NAT for a stable egress IP by @skord in estuary/flow#3044
* docs: document the Exclude Flow Document (no_flow_document) materialization option by @jwhartley in estuary/flow#3018
* oidc-discovery-server: survive direct VPC egress cold-start on deploy by @skord in estuary/flow#3062
* docs: clarify retain_existing_data_on_backfill requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3049
* data-plane-controller: restart on ansiblehost by @qaoj in estuary/flow#3066
* flow-web: Bump version to release by @jshearer in estuary/flow#3068
* ops-catalog: misc updates by @williamhbaker in estuary/flow#3067
* proto-flow: tolerate unknown JSON fields in connector protocols by @williamhbaker in estuary/flow#3059
* flowctl: add raw split-shards to scale out V2 tasks by @williamhbaker in estuary/flow#3021
* supabase: qualify replace_data_plane_releases DELETE for PostgREST by @skord in estuary/flow#3076
* docs: kcat recipes for testing a Dekaf topic by @jwhartley in estuary/flow#3072
* Docs: document per-prefix alert scoping and configurable thresholds by @jwhartley in estuary/flow#3039
* notifications: trial bucket retention is 20 days, not 30 by @jwhartley in estuary/flow#3074
* runtime/container: use ops::decode::Decoder for log lines by @williamhbaker in estuary/flow#3091
* data-plane-controller: remove restart flag by @qaoj in estuary/flow#3087
* runtime-next: durably seed initial connector state as {} to match V1 by @williamhbaker in estuary/flow#3081
* runtime-next: don't abandon the final capture transaction on connector EOF by @jgraettinger in estuary/flow#3090
* Docs: Remove an IP address from GCP allowed list by @jwhartley in estuary/flow#3080
* docs: remove static data plane IP list, reference dashboard by @jwhartley in estuary/flow#3097
* dekaf: improve logging by @williamhbaker in estuary/flow#3078
* agent: make startup logging consistent by @williamhbaker in estuary/flow#3098
* shuffle: make the shuffle disk limit configurable per-task by @jgraettinger in estuary/flow#3096
* runtime-next: automatically split journals under sustained append-rate throttling by @dgreer-dev in estuary/flow#3029
* billing: add tenant billing contact fields and per-tenant controller by @jshearer in estuary/flow#2902
* flowctl-go(api test): raise test-shard readiness window from ~3s to ~30s by @jshearer in estuary/flow#3101
* runtime: strip the V2 committed-close marker for derivations by @williamhbaker in estuary/flow#3100
* deps: bump aws-lc-sys and lz4_flex for security advisories by @skord in estuary/flow#2875
* data-plane-controller: fix db timeout by @qaoj in estuary/flow#3105
* agent-api: survive direct VPC egress cold-start on instance startup by @skord in estuary/flow#3109
* docs: avoiding backfills during a database failover or host change by @jwhartley in estuary/flow#3086
* docs: clarify column-level SELECT grants unsupported for MySQL/MariaDB CDC by @jwhartley in estuary/flow#3085
* json: require RFC3339 'T' separator in date-time format validator by @jacobmarble in estuary/flow#3116
* control-plane: reserve privileged tenant names by @jwhartley in estuary/flow#3083
* agent-api: harden startup and shutdown on Cloud Run by @skord in estuary/flow#3114
* Incidental fixes: gazette Append retry, connector-init image, materialize tear-down by @jgraettinger in estuary/flow#3121
* agent: flag to create new captures as runtime-v2 by @williamhbaker in estuary/flow#3107
* runtime-next: add minimum interval for post-txn triggers by @dgreer-dev in estuary/flow#3110
* flowctl: chunk API requests by url size rather than fixed count by @mdibaiee in estuary/flow#3123
* dekaf: use projection_constraints only, no constraints by @mdibaiee in estuary/flow#3124
* flowctl: normalize --prefix trailing slash to avoid misleading PermissionDenied by @GregorShear in estuary/flow#3075
* docs: retain_existing_data_on_backfill no longer requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3125
* Restart Materializations sessions before IAM token expiry by @dgreer-dev in estuary/flow#3130
* docs: add "bring your own app" instructions for source-quickbooks by @nicolaslazo in estuary/flow#3135
* flowctl: re-tool `preview` on the runtime-next stack by @jgraettinger in estuary/flow#3117
* Adding support for updating the quotas when payment info changes by @bbartman in estuary/flow#3127
* control-plane-api: add unauthenticated publicDataPlanes GraphQL query by @GregorShear in estuary/flow#3129
* go/bindings: update snapshots for materialize-sqlite ser_policy by @dgreer-dev in estuary/flow#3142
* Revert "flowctl: replace `preview` with the runtime-next implementation" by @jgraettinger in estuary/flow#3141
* local: per-checkout stacks — every checkout runs its own isolated stack by @jgraettinger in estuary/flow#3137
* dekaf: log tls handshake eof instead of returning as error by @danielnelson in estuary/flow#3143
* Updating migrations to better support testing. by @bbartman in estuary/flow#3146
* shuffle: raise causal-hint stall timeout to 15m; lower prune horizon to 2GB by @jgraettinger in estuary/flow#3148
* ci: fix Platform Build by packaging Go binaries from per-checkout $GOBIN by @jgraettinger in estuary/flow#3149
* capture+materialize: Support Nonsensitive Field Overlays by @willdonnelly in estuary/flow#3119
* Docs: The Great Description-ing by @aeluce in estuary/flow#3057
* runtime: fix recursive read-lock deadlock in capture-v2 buildJoin by @jgraettinger in estuary/flow#3156
* validation: surface write-schema redact annotations on projections of split-schema collections by @GregorShear in estuary/flow#3147
* control-plane-api: retry Stripe customer search on index-lag misses by @jgraettinger in estuary/flow#3157
* flowctl: multi-shard --fixture for raw preview-next by @mdibaiee in estuary/flow#3154
* dekaf: prevent cached `TimeoutNoData` fetch responses from incorrectly signaling EOF by @jshearer in estuary/flow#3153
* mise: remove Lima VM host share and other VM tweaks by @jgraettinger in estuary/flow#3163
* docs: document connecting Azure Private Link to native Azure resources by @jwhartley in estuary/flow#3138
* docs: explain why exclusiveCollectionFilter needs few enabled bindings by @jwhartley in estuary/flow#3126

## New Contributors
* @bbartman made their first contribution in estuary/flow#3127

**Full Changelog**: estuary/flow@v0.6.10...v0.6.11

Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.