Tags: estuary/flow
Tags
control-plane: add `closed` flag to data planes (#3175) Adds a `closed` boolean to data planes and lets the `dataPlanes` GraphQL query filter on it, so operators can retire a plane from new selection without deleting its record — a closed plane keeps serving existing tasks but drops out of listings. "Closed" reflects that the plane is still fully live; it is only withheld from new selection. - DB: new `data_planes.closed` column, `boolean NOT NULL DEFAULT false` (existing planes stay open), with a `SELECT (closed)` grant to `authenticated`. - GraphQL: `closed: Boolean!` on the `DataPlane` type, plus a `DataPlanesFilter` argument on `dataPlanes` carrying `closed: BoolFilter`. `filter.closed.eq` narrows to matching planes; omitting it returns both open and closed planes. - The flag rides on the authorization snapshot, so both the filter and the per-node value resolve in-memory with no database round-trip. The filter is applied before pagination, keeping page sizes and cursors correct. The snapshot lags the database by up to one refresh interval, which is acceptable for a listing filter. - The unauthenticated `publicDataPlanes` query (pre-signup selection) excludes closed planes unconditionally and takes no filter argument — a retired plane should never be offered to a new signup.
docs: explain why exclusiveCollectionFilter needs few enabled bindings ( #3126) * docs: explain why exclusiveCollectionFilter needs few enabled bindings The description said 'should only be used if a small number of bindings are enabled' without the why. Add it: the filter is evaluated against every oplog event with one clause per enabled binding, so with many bindings it can slow change stream reads enough for lag to grow, and excludeCollections is the better choice. * docs: add exclusiveCollectionFilter explainer to MongoDB variant pages Same one-line addition from the mongodb.md fix, applied to the DocumentDB and Cosmos DB pages -- both run the same source-mongodb connector and carried the identical unqualified caveat. Per aeluce's review on #3126.
docs: document associative: false for disabling key deduplication (#3008 ) * docs: document associative: false for disabling key deduplication * doc: drop EXPERIMENTAL note from associative reduction keyword * docs: clarify which reduction stages associative: false affects * docs: address review — annotation works at any location; include derivations
materialize-clickhouse: new connector (#2776) * materialize-clickhouse: new connector * update per changes to implementation * materialize-clickhouse: document required permissions Add a Required Permissions section with the specific grants needed: GRANT ALL on the target database, plus SELECT on system.columns, system.parts, and system.tables (not covered by the database grant). Also document optional row-level policies for restricting system table visibility to the target database only. * integrate feedback
control-plane-api: enforce strict $defs check in update_connector_schema
The legacy migration path in update_connector_schema silently replaced
any existing schema that didn't mention flow://connector-schema with a
new {$ref: flow://connector-schema} wrapper. This was intended to
migrate old collections into the managed-schema pattern, but that
migration is long complete. Keeping it is a liability: if a user
manually manages their collection schema and accidentally enables
auto-discover, the migration silently blows away their work and may
initialize a read schema under the connector's direction — changes
that are difficult to notice and undo.
Now update_connector_schema requires that existing schemas already
have a $defs entry keyed by flow://connector-schema. Schemas
without the $defs entry get a clear error message with opt-in
instructions.
Also cleans up test infrastructure:
- Extract wrap_connector_schema helper to shared integration test module
- Delete 13 orphaned/stale snapshot files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PreviousNext