Self-service private-link configuration by jshearer · Pull Request #2944 · estuary/flow

jshearer · May 14, 2026

This PR exposes the data_planes.private_links configuration through the GraphQL API. Today private data-plane customers file a support request and we edit the column by hand.

`dataPlanes` query: private-networking fields

privateLinks returns the configured endpoints as a typed union (AWSPrivateLink | AzurePrivateLink | GCPPrivateServiceConnect).
awsLinkEndpoints / azureLinkEndpoints / gcpPscEndpoints expose the provisioning results the data-plane controller writes back, as opaque JSON.
All four fields are gated on the ViewDataPlanePrivateNetworking capability and fail closed to an empty list.

`updateDataPlanePrivateLinks` mutation

Takes [PrivateLinkInput!]!, a oneof input mirroring the union, and replaces the entire private_links array on a private data plane; partial updates are intentionally not supported. The DPC converges to the new configuration on its next poll.
Requires ModifyDataPlanePrivateNetworking on the data-plane name, after a structural check that the name is under ops/dp/private/<tenant>/.

Authorization model

Two new capability bits: ViewDataPlanePrivateNetworking and ModifyDataPlanePrivateNetworking.
The Viewer bundle (and therefore legacy read) carries the View bit: read on a data-plane prefix already conveys deploy-level trust, so viewing the plane's networking configuration comes with it.
Modify reaches a tenant admin through the conjunction of two grants: the ops/dp/private/<tenant>/ role_grant must carry the new ManageDataPlane bundle (create_data_plane installs it at provisioning time; a migration backfills existing private data planes), and the user's own bits at the tenant must include the bit, which today means admin since Admin folds ManageDataPlane in. Bits intersect along the delegation chain, so neither grant alone conveys Modify.
To grant ModifyDataPlanePrivateNetworking to a non-admin, create a user_grant on the tenant carrying {manage_data_plane, delegate} (Editor already carries Delegate, so editors need only manage_data_plane).

Shared models

The PrivateLink types moved from data-plane-controller::shared::stack to models, with feature-gated async-graphql derives so one set of structs backs the DB column, the GraphQL schema, and the DPC (which re-exports them).
Azure's dns_name / resource_type are now Option<String>. The previous shape was String with skip_serializing_if = "String::is_empty", so "" and absent were already indistinguishable on the wire: both serialized as a missing field. The new deserializer normalizes an incoming "" to None, and serialization stays field-absent, so the stored bytes and the stack config est-dry-dock reads are identical for every historical row shape (absent, "", or a value); a round-trip test in models::private_links pins this.
est-dry-dock cannot observe the difference either: its Pydantic models default these fields to "" and use truthiness checks. The visible effect is confined to GraphQL, where the fields are nullable and rows that stored "" surface as null.

Reviewer notes

Trade-offs discussed during review, recorded so they don't need to be re-litigated:

The four private-networking resolvers are deliberately not deduplicated further. The non-trivial shared piece, the capability check, is the authorized() helper in graphql/mod.rs; what remains per resolver is the gate call plus a clone-and-wrap body. A gated_endpoints helper bundling the gate and the JSON mapping was tried and dropped as trivial-body indirection under a vague name. Keeping the gate visible inside each resolver keeps the authorization audit local; the cost is that a change to the denial behavior must be edited in four places.
Azure Option<String> was chosen over two alternatives: keeping String (zero glue, but the "" sentinel becomes part of the public GraphQL contract as String!), and mapping "" to null at the GraphQL layer only (DPC untouched, but it requires an Azure-only split of input and output types while the model keeps the sentinel). Normalizing in models costs a small serde adapter plus a round-trip test, and makes the nullable contract hold everywhere the type is used.
ViewDataPlanePrivateNetworking is kept as its own bit even though read currently implies it via Viewer. The rejected alternative was to drop the bit and let DP-node visibility be the gate, which would have deleted the field gates entirely. Keeping the bit means each field gate names its capability explicitly, and the View / Modify split survives future changes to what read implies.

Deployment note

Migrations are applied manually; the sequencing matters:

Apply the enum migration before deploying: the new create_data_plane binds the manage_data_plane value, so provisioning a private data plane on the new code errors if the value is missing. (Two migration files because alter type add value cannot be referenced in the same transaction.)
Run the backfill only after the rollout completes: binaries built before this PR decode bundles strictly in the snapshot loader, so any row carrying manage_data_plane breaks their role_grants fetch. For the same reason, do not provision a private data plane mid-rollout while old pods are live.
In the window between deploy and backfill, tenant admins of existing private data planes can view private links but not modify them. The backfill is idempotent and skips rows that already carry the bundle, so it is safe to run any time after the rollout, and planes provisioned on the new code are unaffected by it.

Fixes #2773

…async-graphql derives The GraphQL API and the data-plane controller need to speak the same `PrivateLink` shape: DPC reads the `private_links json[]` column off `data_planes` and ships it into the est-dry-dock Pulumi stack, and the GraphQL surface needs both typed read and a typed input for the mutation that replaces the column. Defining the types once in `models` is the only way to share them without making DPC depend on the GraphQL crate. * Move `PrivateLink`, `AWSPrivateLink`, `AzurePrivateLink`, `GCPPrivateServiceConnect` to `models::private_links`, re-export from `data-plane-controller::shared::stack` so existing DPC callers keep compiling unchanged. * Add async-graphql derives under the existing `async-graphql` feature: `Union` + `OneofObject` on `PrivateLink`, `SimpleObject` + `InputObject` on each provider struct. Same Rust enum drives `union PrivateLink` on output and `input PrivateLinkInput @oneOf` on input, so the two GraphQL surfaces cannot drift. * Azure `dns_name` and `resource_type` flip from `String`-with-empty-as-sentinel to `Option<String>` with a serde adapter that normalizes both an absent field and an empty string to `None` on read and omits both `None` and `Some("")` on write. Wire format byte-for-byte unchanged; the GraphQL surface exposes them as nullable instead of requiring clients to know the empty-string convention. * AWS `service_region: Option<String>` carries the master-side change (commit b1d51ed) into the moved type, with `skip_serializing_if = "Option::is_none"` so the column stays absent (not `null`) when unset, preserving est-dry-dock's existing fallback to `region`.

…d `updateDataPlanePrivateLinks` mutation Exposes the configured private-link endpoints on the `dataPlanes` query as a typed union against `models::PrivateLink`, alongside the three provisioning-result endpoint columns (`awsLinkEndpoints`, `azureLinkEndpoints`, `gcpPscEndpoints`) as opaque JSON arrays. Adds a mutation that replaces the entire `private_links` column on a private data plane; the data-plane controller picks up the new configuration on its next poll and converges via Pulumi. * `dataPlanes` selects the raw `private_links` JSON once per page and parses each entry lazily in a `ComplexObject` resolver, so a malformed historical row produces a descriptive field-level error naming the data plane and the failing index rather than breaking the whole query selection. * `updateDataPlanePrivateLinks(dataPlaneName, privateLinks)` accepts `[PrivateLinkInput!]!` directly against the same `models::PrivateLink` types via async-graphql's `@oneOf` input; supplying multiple branches in a single element is rejected by GraphQL validation before the handler runs. * Name gating is a structural check: the name must sit under `ops/dp/private/<tenant>/...`. Anything more specific (cluster suffix shape, owning prefix shape) is the data plane's problem; an unknown name falls out as "not found" when the UPDATE matches zero rows. Per-provider validation is intentionally not duplicated here: the authoritative schema is the est-dry-dock Pydantic model on the consumer side, and a bad shape surfaces via DPC convergence. * Interim authorization requires `read` on the private data-plane name, mirroring the existing deployment mutation shape. This will move to `manage_dataplane` once the orthogonal capability model lands. * Returns `Boolean!` so the mutation surface is minimal; callers that need the post-write state re-query `dataPlanes`.

Adds two capability bits, `ViewDataPlanePrivateNetworking` and `ModifyDataPlanePrivateNetworking`, composed by a new `ManageDataPlane` bundle. The bundle is granted on the role-grant edge `<tenant>/ -> ops/dp/private/<tenant>/` at provisioning time, so tenant admins (whose `Admin` bundle composes `ManageDataPlane`) inherit both bits on their private data planes; non-admin tenant grantees intersect to their own bundle at the edge and never reach these bits. * `private_links` resolver on `DataPlane` returns an empty list to callers that lack `ViewDataPlanePrivateNetworking` on the data plane name. * `updateDataPlanePrivateLinks` mutation requires `ModifyDataPlanePrivateNetworking` (replacing the interim legacy `read` check). * `evaluate_names_authorization` accepts any `Into<CapabilitySet> + Display + Copy`, letting call sites pass either legacy `models::Capability` or a single `models::authz::Capability` bit; legacy callers keep working unchanged via the `From<legacy>` impl. * `models::authz::Capability` gets a `Display` impl that delegates to `Debug`, so error messages reference each bit by its Rust identifier. * `create_data_plane.rs` stamps `bundles = '{manage_data_plane}'` on the new role grant. The accompanying two-step migration adds the enum value and backfills existing rows -- `alter type add value` cannot be used in the same transaction that subsequently references the new value.

…lities The service-account management surface was asymmetric: listing authorized on the fine-grained `ManageServiceAccounts` capability while every mutation required the full `Admin` bundle. That produced a class of principal (a `TeamAdmin` without `Admin`) who could see service accounts but manage none of them, and it meant the named capability the feature defines was never the actual gate for any write. Make the surface track the capabilities it defines: - Anchor checks (create/add-grant/remove-grant/create-key/revoke-key, all on the account's `catalog_name`) now require `ManageServiceAccounts`, matching the listing query. A `TeamAdmin` can fully manage accounts and their keys without holding `Admin`. - The per-grant prefix check now requires `CreateGrant` on each granted prefix rather than `Admin`. This is the anti-escalation guard — a caller still can't hand a service account reach they couldn't grant anyone — but it keys off the capability that authorizes granting, not the whole Admin bundle. Human-user grant creation still lives in PostgREST; when it migrates to GraphQL it should adopt this same `CreateGrant` gate. To express fine-grained capabilities at these call sites, generalize `evaluate_names_authorization` and `verify_authorization` to accept anything that converts into a `CapabilitySet` (legacy `models::Capability`, a single `models::authz::Capability` bit, or an explicit set). The BFS primitive already operated on `CapabilitySet`, so this only lifts the wrapper signatures; existing legacy-capability callers are unaffected. Add a `Display` impl for `models::authz::Capability` so denial messages render the required capability. These three changes mirror the same generalization in #2944 so the two branches converge cleanly on whichever merges second. Add `test_team_admin_manages_without_full_admin`, which seeds a caller holding only the `team_admin` bundle (no `Admin`) and asserts both the positive path (manage accounts, mint keys, grant prefixes within reach) and the anti-escalation boundary (cannot grant a prefix they lack `CreateGrant` on). Also correct the migration comments, which claimed API keys are never exchanged for a JWT — the token-exchange endpoint mints a short-lived access token from a key.

## What's Changed * runtime: periodically store primary FSMHints of V2 shards by @williamhbaker in estuary/flow#3027 * mise build:flowctl for a local flowctl binary by @mdibaiee in estuary/flow#3031 * go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3033 * data-plane-controller: dns and s3 passthroughs by @qaoj in estuary/flow#3022 * Self-service private-link configuration by @jshearer in estuary/flow#2944 * supabase: scoped refresh tokens so CI writes via PostgREST instead of direct psql by @skord in estuary/flow#3013 * docs: Describe SQL Server replica suport by @willdonnelly in estuary/flow#3040 * proto-gazette: regenerate to pick up SUSPEND_KEEP by @williamhbaker in estuary/flow#3041 * docs: correct custom-column-types backfill behavior for DDL changes by @jwhartley in estuary/flow#3036 * Refresh token GraphQL operations and token exchange endpoint by @GregorShear in estuary/flow#3020 * docs: shopify stream & resource config additions by @Alex-Bair in estuary/flow#3034 * validation/collection: exclude `flow://inferred-schema` from the managed-defs redact check by @jshearer in estuary/flow#3046 * docs: update Bigtable connector permissions example by @mwillman-estuary in estuary/flow#3045 * docs: disable auto-discover for multi-binding file source captures by @jwhartley in estuary/flow#3037 * docs: add streams and API pinning details to stripe-native by @nicolaslazo in estuary/flow#3055 * data-plane-controller: restart setting by @qaoj in estuary/flow#3048 * deploy: route control-plane services through the VPC NAT for a stable egress IP by @skord in estuary/flow#3044 * docs: document the Exclude Flow Document (no_flow_document) materialization option by @jwhartley in estuary/flow#3018 * oidc-discovery-server: survive direct VPC egress cold-start on deploy by @skord in estuary/flow#3062 * docs: clarify retain_existing_data_on_backfill requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3049 * data-plane-controller: restart on ansiblehost by @qaoj in estuary/flow#3066 * flow-web: Bump version to release by @jshearer in estuary/flow#3068 * ops-catalog: misc updates by @williamhbaker in estuary/flow#3067 * proto-flow: tolerate unknown JSON fields in connector protocols by @williamhbaker in estuary/flow#3059 * flowctl: add raw split-shards to scale out V2 tasks by @williamhbaker in estuary/flow#3021 * supabase: qualify replace_data_plane_releases DELETE for PostgREST by @skord in estuary/flow#3076 * docs: kcat recipes for testing a Dekaf topic by @jwhartley in estuary/flow#3072 * Docs: document per-prefix alert scoping and configurable thresholds by @jwhartley in estuary/flow#3039 * notifications: trial bucket retention is 20 days, not 30 by @jwhartley in estuary/flow#3074 * runtime/container: use ops::decode::Decoder for log lines by @williamhbaker in estuary/flow#3091 * data-plane-controller: remove restart flag by @qaoj in estuary/flow#3087 * runtime-next: durably seed initial connector state as {} to match V1 by @williamhbaker in estuary/flow#3081 * runtime-next: don't abandon the final capture transaction on connector EOF by @jgraettinger in estuary/flow#3090 * Docs: Remove an IP address from GCP allowed list by @jwhartley in estuary/flow#3080 * docs: remove static data plane IP list, reference dashboard by @jwhartley in estuary/flow#3097 * dekaf: improve logging by @williamhbaker in estuary/flow#3078 * agent: make startup logging consistent by @williamhbaker in estuary/flow#3098 * shuffle: make the shuffle disk limit configurable per-task by @jgraettinger in estuary/flow#3096 * runtime-next: automatically split journals under sustained append-rate throttling by @dgreer-dev in estuary/flow#3029 * billing: add tenant billing contact fields and per-tenant controller by @jshearer in estuary/flow#2902 * flowctl-go(api test): raise test-shard readiness window from ~3s to ~30s by @jshearer in estuary/flow#3101 * runtime: strip the V2 committed-close marker for derivations by @williamhbaker in estuary/flow#3100 * deps: bump aws-lc-sys and lz4_flex for security advisories by @skord in estuary/flow#2875 * data-plane-controller: fix db timeout by @qaoj in estuary/flow#3105 * agent-api: survive direct VPC egress cold-start on instance startup by @skord in estuary/flow#3109 * docs: avoiding backfills during a database failover or host change by @jwhartley in estuary/flow#3086 * docs: clarify column-level SELECT grants unsupported for MySQL/MariaDB CDC by @jwhartley in estuary/flow#3085 * json: require RFC3339 'T' separator in date-time format validator by @jacobmarble in estuary/flow#3116 * control-plane: reserve privileged tenant names by @jwhartley in estuary/flow#3083 * agent-api: harden startup and shutdown on Cloud Run by @skord in estuary/flow#3114 * Incidental fixes: gazette Append retry, connector-init image, materialize tear-down by @jgraettinger in estuary/flow#3121 * agent: flag to create new captures as runtime-v2 by @williamhbaker in estuary/flow#3107 * runtime-next: add minimum interval for post-txn triggers by @dgreer-dev in estuary/flow#3110 * flowctl: chunk API requests by url size rather than fixed count by @mdibaiee in estuary/flow#3123 * dekaf: use projection_constraints only, no constraints by @mdibaiee in estuary/flow#3124 * flowctl: normalize --prefix trailing slash to avoid misleading PermissionDenied by @GregorShear in estuary/flow#3075 * docs: retain_existing_data_on_backfill no longer requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3125 * Restart Materializations sessions before IAM token expiry by @dgreer-dev in estuary/flow#3130 * docs: add "bring your own app" instructions for source-quickbooks by @nicolaslazo in estuary/flow#3135 * flowctl: re-tool `preview` on the runtime-next stack by @jgraettinger in estuary/flow#3117 * Adding support for updating the quotas when payment info changes by @bbartman in estuary/flow#3127 * control-plane-api: add unauthenticated publicDataPlanes GraphQL query by @GregorShear in estuary/flow#3129 * go/bindings: update snapshots for materialize-sqlite ser_policy by @dgreer-dev in estuary/flow#3142 * Revert "flowctl: replace `preview` with the runtime-next implementation" by @jgraettinger in estuary/flow#3141 * local: per-checkout stacks — every checkout runs its own isolated stack by @jgraettinger in estuary/flow#3137 * dekaf: log tls handshake eof instead of returning as error by @danielnelson in estuary/flow#3143 * Updating migrations to better support testing. by @bbartman in estuary/flow#3146 * shuffle: raise causal-hint stall timeout to 15m; lower prune horizon to 2GB by @jgraettinger in estuary/flow#3148 * ci: fix Platform Build by packaging Go binaries from per-checkout $GOBIN by @jgraettinger in estuary/flow#3149 * capture+materialize: Support Nonsensitive Field Overlays by @willdonnelly in estuary/flow#3119 * Docs: The Great Description-ing by @aeluce in estuary/flow#3057 * runtime: fix recursive read-lock deadlock in capture-v2 buildJoin by @jgraettinger in estuary/flow#3156 * validation: surface write-schema redact annotations on projections of split-schema collections by @GregorShear in estuary/flow#3147 * control-plane-api: retry Stripe customer search on index-lag misses by @jgraettinger in estuary/flow#3157 * flowctl: multi-shard --fixture for raw preview-next by @mdibaiee in estuary/flow#3154 * dekaf: prevent cached `TimeoutNoData` fetch responses from incorrectly signaling EOF by @jshearer in estuary/flow#3153 * mise: remove Lima VM host share and other VM tweaks by @jgraettinger in estuary/flow#3163 * docs: document connecting Azure Private Link to native Azure resources by @jwhartley in estuary/flow#3138 * docs: explain why exclusiveCollectionFilter needs few enabled bindings by @jwhartley in estuary/flow#3126 ## New Contributors * @bbartman made their first contribution in estuary/flow#3127 **Full Changelog**: estuary/flow@v0.6.10...v0.6.11 Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>

jshearer force-pushed the jshearer/private_links_gql branch from d41b7a5 to e9736cb Compare May 27, 2026 21:14

jshearer changed the base branch from master to greg/authz/1 May 27, 2026 21:14

jshearer force-pushed the jshearer/private_links_gql branch from 4a99da9 to 6ab0520 Compare May 27, 2026 21:50

jshearer force-pushed the greg/authz/1 branch from 164cd1b to 3d0aa54 Compare May 28, 2026 14:34

jshearer changed the base branch from greg/authz/1 to master May 28, 2026 18:38

jshearer force-pushed the jshearer/private_links_gql branch from 6ab0520 to 55164df Compare May 28, 2026 18:41

jshearer force-pushed the jshearer/private_links_gql branch from 6ec128a to 5bf9a1f Compare June 10, 2026 21:25

jshearer changed the title ~~Privatelinks GraphQL~~ Self-service private-link configuration Jun 10, 2026

jshearer marked this pull request as ready for review June 10, 2026 22:22

jshearer requested a review from GregorShear June 10, 2026 22:22

GregorShear reviewed Jun 12, 2026

View reviewed changes

jshearer force-pushed the jshearer/private_links_gql branch from 5bf9a1f to 6e4b40c Compare June 15, 2026 20:01

jshearer force-pushed the jshearer/private_links_gql branch from 6e4b40c to 9c34298 Compare June 15, 2026 20:38

jshearer added 2 commits June 15, 2026 19:20

jshearer force-pushed the jshearer/private_links_gql branch from 9c34298 to 4095dbd Compare June 15, 2026 23:21

jshearer requested a review from GregorShear June 15, 2026 23:37

GregorShear approved these changes Jun 16, 2026

View reviewed changes

jshearer merged commit e63dfe3 into master Jun 16, 2026
15 of 17 checks passed

github-actions Bot mentioned this pull request Jul 15, 2026

🚀 Release 0.6.11 estuary/homebrew-flowctl#51

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Self-service private-link configuration - #2944

Self-service private-link configuration#2944
jshearer merged 3 commits into
masterestuary/flow:masterfrom
jshearer/private_links_gqlestuary/flow:jshearer/private_links_gqlCopy head branch name to clipboard

jshearer commented May 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

jshearer commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

dataPlanes query: private-networking fields

updateDataPlanePrivateLinks mutation

Authorization model

Shared models

Reviewer notes

Deployment note

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jshearer commented May 14, 2026 •

edited

Loading

`dataPlanes` query: private-networking fields

`updateDataPlanePrivateLinks` mutation