flowctl: re-tool preview on the runtime-next stack - #3117
#3117Merged
Merged
flowctl: re-tool preview on the runtime-next stack#3117
preview on the runtime-next stack#3117Conversation
preview on the runtime-next stack
jgraettinger
force-pushed
the
johnny/preview
branch
5 times, most recently
from
July 6, 2026 16:34
f6adb49 to
53cacd6
Compare
jgraettinger
marked this pull request as ready for review
July 6, 2026 18:21
| schema, | ||
| .. | ||
| } => { | ||
| fields.push(("collection", json_field(collection_name))); |
Member
There was a problem hiding this comment.
Wanting to confirm this field name...should it be collection_name? Since the L1 inferred schemas rollup is looking for that:
Member
Author
There was a problem hiding this comment.
Yes good catch. I'll fix, but I'd also like to work schema inference into the soak test so it provides a positive verification it's working.
| // The patch payload is valid JSON (a tab-delimited JSON array; | ||
| // see `crate::patches`), so it embeds verbatim. | ||
| fields.push(("patches", persist.connector_patches_json.clone())); | ||
| (ops::LogLevel::Debug, "persisted connector-state delta") |
Member
There was a problem hiding this comment.
Will these get filtered out for info and above level tasks, since they don't go through Tokio's tracing subscriber?
Member
Author
There was a problem hiding this comment.
Yes. We're paying a bit of cost that tracing avoids, in that we're actually building the structured event to have it then be filtered by tracing at point of emission. I wasn't concerned though (Bytes ref-count clone, or bare &ref)
jgraettinger
force-pushed
the
johnny/preview
branch
from
July 7, 2026 22:05
53cacd6 to
c170173
Compare
Member
Author
|
PTAL -- fixed the schema inference regression, and added new coverage in the soak test to provide E2E verification. |
…tories Factor runtime-next's host-specific behavior — how a task publishes, shuffles, and logs — out of the crate into three uniform, monomorphized host seams, each a factory the leader and shard `Service`s are generic over: - publish: replace the `Publisher` enum — a `Real` variant plus a `Preview` variant that baked NDJSON-to-stdout output into the crate — with a `PublisherFactory` / `JournalPublisher(Factory)` seam. The publisher now carries only document output; its former observation calls move to the Logger seam. - leader: factor the leader's inline shuffle-`Session` opening and checkpoint sourcing into a new `leader/shuffle.rs`, providing `ShuffleSession` / `ShuffleSessionFactory` (static RPITIT, not `dyn`) plus a `ShuffleServiceFactory`. - log: new `Logger` / `LoggerFactory` seam — the task's out-of-band log and event stream. A `Logger` sinks both the connector's own logs (`log`) and structured runtime `LogEvent`s (`event`): a `#[non_exhaustive]` enum (persist / applied / inferred-schema / container lifecycle) whose variants carry borrowed, verbatim payloads so a host may intercept them structurally. Its canonical `LogEvent::to_log` flattens each event into the ops-log line it replaced, and `event` defaults to that flattening — so a host overrides only to intercept, delegating the rest. Production shards install `FnLoggerFactory` (sinking to the task-log writer); leaders and tests install `TracingLoggerFactory`. Remove the now-unnecessary `Task.preview` flag from the runtime proto (reserving field 2): how a task publishes and logs is a host concern decided by the installed factories, not a protocol flag. `max_transactions` stays as the transaction bound, re-documented as a general preview/harness control rather than tied to the removed flag. Make the `shard::rocksdb` module public — exposing `RocksDB` (with its `open`, `scan`, and now-public `put_connector_state_base`) and re-exporting its `DecodeError` — so a host harness can seed shard zero's RocksDB by path and read back its final reduced state itself, and re-expose `shuffle::log::BlockMeta` for harness segment writers.
Drive `flowctl raw preview-next` through runtime-next's three factory seams by installing preview-specific implementations of each: - publish: an output-capturing `PreviewPublisherFactory` writes captured / derived documents to stdout as NDJSON. - log: a `PreviewLoggerFactory` sinks the task's log stream and powers `--output-state` / `--output-apply` (the legacy `connectorState` / `applied` lines) by intercepting runtime-next's `LogEvent::Persist` / `LogEvent::Applied` structurally — decoding state deltas from its tab-framed patch wire form — and flattening every other event through `LogEvent::to_log` into the same log handler as the connector's own logs. The `--output-state` run-end final reduced state is emitted by re-opening shard zero's RocksDB (`read_preview_state`) once the session loop has released it. - shuffle: a `PreviewShuffleFactory` enum selects per run between a live in-process journal-reading shuffle Session and a `--fixture` replay (`FixtureOpener`, a `ShuffleSessionFactory` that writes log segments and feeds synthetic checkpoint Frontiers). `--initial-state` seeds shard zero's RocksDB by path, and `--output-state`'s run-end read reopens it — both managed by preview itself through the now-public `runtime_next::shard::rocksdb` handle (the local `seed_preview_state` / `read_preview_state` helpers over `RocksDB::open`, `put_connector_state_base`, and `scan`). Each driver also encodes its task spec once and threads the shared bytes to its shards rather than re-encoding per shard. `--delay`, eager/streaming fixtures, and `--sessions` planning are unchanged.
`flowctl preview` now runs on the runtime-next + shuffle stack that was
developed on this branch as `flowctl raw preview-next`, which the prior
commits verified to be behaviorally compatible with the legacy preview.
Move the `raw/preview_next/` module into `preview/`, delete the old
V1-runtime `preview/{mod,journal_reader}.rs`, and drop the `raw
preview-next` command wiring. The CLI flag surface, stdout/stderr line
shapes, and exit-code contract that the estuary/connectors CI harnesses
depend on are preserved; `--shards` and `--debug-port` are additive.
The old preview was the sole consumer of `runtime::harness` (and its
private `exchange` combinator), so remove that now-dead module too.
Update the soak README to the new command name, and delete the
runtime-v2 preview harness guide: a closed, point-in-time document
describing `raw preview-next` as it stood during bring-up.
Gate the soak's downstream chain on schema inference so it provides
continuous, positive proof that inference works — the failure mode that
motivated this (a silently mis-keyed inferred-schema ops log) presents as
`accounts` never coming up, which the previous fixed-schema soak missed.
- events/{alpha,beta,gamma} take an inferred readSchema
(allOf: [flow://write-schema, flow://inferred-schema]); events.schema.json
stays the writeSchema / wire contract.
- source-soak emits a partial SourcedSchema per binding: id/ts/set/oracle are
sourced and stable; seq/balanceDelta/transfer are left to doc-driven
inference, so their bounds churn under load while the existing exactly-once
and conservation probes ride through each spec-update-driven restart.
- accounts keeps its explicit schema (a derivation has no SourcedSchema, and
seeded from a complete schema it never widens → no inferred_schemas row,
expected). README/derivation comments document this.
No runtime changes.
jgraettinger
force-pushed
the
johnny/preview
branch
from
July 8, 2026 19:22
c170173 to
fc45cf3
Compare
mdibaiee
added a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Jul 15, 2026
## What's Changed * runtime: periodically store primary FSMHints of V2 shards by @williamhbaker in estuary/flow#3027 * mise build:flowctl for a local flowctl binary by @mdibaiee in estuary/flow#3031 * go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3033 * data-plane-controller: dns and s3 passthroughs by @qaoj in estuary/flow#3022 * Self-service private-link configuration by @jshearer in estuary/flow#2944 * supabase: scoped refresh tokens so CI writes via PostgREST instead of direct psql by @skord in estuary/flow#3013 * docs: Describe SQL Server replica suport by @willdonnelly in estuary/flow#3040 * proto-gazette: regenerate to pick up SUSPEND_KEEP by @williamhbaker in estuary/flow#3041 * docs: correct custom-column-types backfill behavior for DDL changes by @jwhartley in estuary/flow#3036 * Refresh token GraphQL operations and token exchange endpoint by @GregorShear in estuary/flow#3020 * docs: shopify stream & resource config additions by @Alex-Bair in estuary/flow#3034 * validation/collection: exclude `flow://inferred-schema` from the managed-defs redact check by @jshearer in estuary/flow#3046 * docs: update Bigtable connector permissions example by @mwillman-estuary in estuary/flow#3045 * docs: disable auto-discover for multi-binding file source captures by @jwhartley in estuary/flow#3037 * docs: add streams and API pinning details to stripe-native by @nicolaslazo in estuary/flow#3055 * data-plane-controller: restart setting by @qaoj in estuary/flow#3048 * deploy: route control-plane services through the VPC NAT for a stable egress IP by @skord in estuary/flow#3044 * docs: document the Exclude Flow Document (no_flow_document) materialization option by @jwhartley in estuary/flow#3018 * oidc-discovery-server: survive direct VPC egress cold-start on deploy by @skord in estuary/flow#3062 * docs: clarify retain_existing_data_on_backfill requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3049 * data-plane-controller: restart on ansiblehost by @qaoj in estuary/flow#3066 * flow-web: Bump version to release by @jshearer in estuary/flow#3068 * ops-catalog: misc updates by @williamhbaker in estuary/flow#3067 * proto-flow: tolerate unknown JSON fields in connector protocols by @williamhbaker in estuary/flow#3059 * flowctl: add raw split-shards to scale out V2 tasks by @williamhbaker in estuary/flow#3021 * supabase: qualify replace_data_plane_releases DELETE for PostgREST by @skord in estuary/flow#3076 * docs: kcat recipes for testing a Dekaf topic by @jwhartley in estuary/flow#3072 * Docs: document per-prefix alert scoping and configurable thresholds by @jwhartley in estuary/flow#3039 * notifications: trial bucket retention is 20 days, not 30 by @jwhartley in estuary/flow#3074 * runtime/container: use ops::decode::Decoder for log lines by @williamhbaker in estuary/flow#3091 * data-plane-controller: remove restart flag by @qaoj in estuary/flow#3087 * runtime-next: durably seed initial connector state as {} to match V1 by @williamhbaker in estuary/flow#3081 * runtime-next: don't abandon the final capture transaction on connector EOF by @jgraettinger in estuary/flow#3090 * Docs: Remove an IP address from GCP allowed list by @jwhartley in estuary/flow#3080 * docs: remove static data plane IP list, reference dashboard by @jwhartley in estuary/flow#3097 * dekaf: improve logging by @williamhbaker in estuary/flow#3078 * agent: make startup logging consistent by @williamhbaker in estuary/flow#3098 * shuffle: make the shuffle disk limit configurable per-task by @jgraettinger in estuary/flow#3096 * runtime-next: automatically split journals under sustained append-rate throttling by @dgreer-dev in estuary/flow#3029 * billing: add tenant billing contact fields and per-tenant controller by @jshearer in estuary/flow#2902 * flowctl-go(api test): raise test-shard readiness window from ~3s to ~30s by @jshearer in estuary/flow#3101 * runtime: strip the V2 committed-close marker for derivations by @williamhbaker in estuary/flow#3100 * deps: bump aws-lc-sys and lz4_flex for security advisories by @skord in estuary/flow#2875 * data-plane-controller: fix db timeout by @qaoj in estuary/flow#3105 * agent-api: survive direct VPC egress cold-start on instance startup by @skord in estuary/flow#3109 * docs: avoiding backfills during a database failover or host change by @jwhartley in estuary/flow#3086 * docs: clarify column-level SELECT grants unsupported for MySQL/MariaDB CDC by @jwhartley in estuary/flow#3085 * json: require RFC3339 'T' separator in date-time format validator by @jacobmarble in estuary/flow#3116 * control-plane: reserve privileged tenant names by @jwhartley in estuary/flow#3083 * agent-api: harden startup and shutdown on Cloud Run by @skord in estuary/flow#3114 * Incidental fixes: gazette Append retry, connector-init image, materialize tear-down by @jgraettinger in estuary/flow#3121 * agent: flag to create new captures as runtime-v2 by @williamhbaker in estuary/flow#3107 * runtime-next: add minimum interval for post-txn triggers by @dgreer-dev in estuary/flow#3110 * flowctl: chunk API requests by url size rather than fixed count by @mdibaiee in estuary/flow#3123 * dekaf: use projection_constraints only, no constraints by @mdibaiee in estuary/flow#3124 * flowctl: normalize --prefix trailing slash to avoid misleading PermissionDenied by @GregorShear in estuary/flow#3075 * docs: retain_existing_data_on_backfill no longer requires allow_existing_tables_for_new_bindings by @jwhartley in estuary/flow#3125 * Restart Materializations sessions before IAM token expiry by @dgreer-dev in estuary/flow#3130 * docs: add "bring your own app" instructions for source-quickbooks by @nicolaslazo in estuary/flow#3135 * flowctl: re-tool `preview` on the runtime-next stack by @jgraettinger in estuary/flow#3117 * Adding support for updating the quotas when payment info changes by @bbartman in estuary/flow#3127 * control-plane-api: add unauthenticated publicDataPlanes GraphQL query by @GregorShear in estuary/flow#3129 * go/bindings: update snapshots for materialize-sqlite ser_policy by @dgreer-dev in estuary/flow#3142 * Revert "flowctl: replace `preview` with the runtime-next implementation" by @jgraettinger in estuary/flow#3141 * local: per-checkout stacks — every checkout runs its own isolated stack by @jgraettinger in estuary/flow#3137 * dekaf: log tls handshake eof instead of returning as error by @danielnelson in estuary/flow#3143 * Updating migrations to better support testing. by @bbartman in estuary/flow#3146 * shuffle: raise causal-hint stall timeout to 15m; lower prune horizon to 2GB by @jgraettinger in estuary/flow#3148 * ci: fix Platform Build by packaging Go binaries from per-checkout $GOBIN by @jgraettinger in estuary/flow#3149 * capture+materialize: Support Nonsensitive Field Overlays by @willdonnelly in estuary/flow#3119 * Docs: The Great Description-ing by @aeluce in estuary/flow#3057 * runtime: fix recursive read-lock deadlock in capture-v2 buildJoin by @jgraettinger in estuary/flow#3156 * validation: surface write-schema redact annotations on projections of split-schema collections by @GregorShear in estuary/flow#3147 * control-plane-api: retry Stripe customer search on index-lag misses by @jgraettinger in estuary/flow#3157 * flowctl: multi-shard --fixture for raw preview-next by @mdibaiee in estuary/flow#3154 * dekaf: prevent cached `TimeoutNoData` fetch responses from incorrectly signaling EOF by @jshearer in estuary/flow#3153 * mise: remove Lima VM host share and other VM tweaks by @jgraettinger in estuary/flow#3163 * docs: document connecting Azure Private Link to native Azure resources by @jwhartley in estuary/flow#3138 * docs: explain why exclusiveCollectionFilter needs few enabled bindings by @jwhartley in estuary/flow#3126 ## New Contributors * @bbartman made their first contribution in estuary/flow#3127 **Full Changelog**: estuary/flow@v0.6.10...v0.6.11 Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
flowctl previewnow runs on the runtime-next + shuffle stack instead of thelegacy V1
runtime::harness. It was developed on this branch asflowctl raw preview-next, then promoted to be thepreview; the V1 preview harness — thelast consumer of
runtime::harness— is deleted along with the now-deadruntime/src/harness/*and its privateexchangecombinator.The user-facing contract is unchanged: the CLI flags, stdout/stderr line shapes,
and exit codes that
estuary/connectorsCI depends on.--shardsand--debug-portare new, additive flags.To get there, runtime-next's host-specific behavior — how a task publishes
documents, reads its sources, and emits logs — was factored into three factory
seams. Production installs journal-backed factories;
previewinstallsoutput-capturing / fixture-replay factories from
flowctl. runtime-next is nowpreview-agnostic, so the lone
Task.previewflag — the "write documents tooutput instead of collections" wire control — is removed from the runtime proto
(field 2 reserved); how a task publishes and logs is now a host concern of the
installed factories, not a wire flag. (
max_transactionsstays as thetransaction bound, re-documented as a general preview/harness control.)
The three host seams (
runtime-next)Each seam is a factory the leader and shard
Services are generic over (staticRPITIT, not
dyn):PublisherFactory/JournalPublisherseam replaces thePublisherenum (aRealvariant plus aPreviewvariant that bakedNDJSON-to-stdout output into the crate); the publisher now carries only
document output, with its former observation calls moved to the logger seam.
leader/shuffle.rsprovidesShuffleSession(Factory)ShuffleServiceFactory, factoring out the leader's previously-inlineshuffle-
Sessionopening and checkpoint sourcing.Logger/LoggerFactoryseam sinking both connector logs(
log) and structuredLogEvents (event) — a#[non_exhaustive]enum(persist / applied / inferred-schema / container lifecycle) carrying borrowed,
verbatim payloads so a host can intercept them structurally.
LogEvent::to_logflattens each event into the ops-log line it replaced, and
eventdefaults tothat flattening, so a host overrides only the events it cares about. Production
shards install
FnLoggerFactory; leaders and tests installTracingLoggerFactory.New harness helpers:
seed_initial_connector_state/read_final_connector_stateseed and read shard zero's RocksDB by path;
shuffle::log::BlockMetaisre-exposed for segment writers.
Preview on the seams (
flowctl)PreviewPublisherFactorywrites captured/derived documents tostdout as NDJSON.
PreviewLoggerFactorysinks the task log and powers--output-state/
--output-apply(the legacyconnectorState/appliedlines) byintercepting
LogEvent::Persist/LogEvent::Appliedstructurally andflattening everything else through
LogEvent::to_log. The run-end final reducedstate comes from re-opening shard zero's RocksDB once the session loop releases
it.
PreviewShuffleFactoryselects per run between a live in-processjournal-reading
Sessionand a--fixturereplay (FixtureOpener).--initial-stateseeds shard zero's RocksDB viaseed_initial_connector_state.--delay, fixtures, and--sessionsplanning are unchanged.Tests / soak
plans/runtime-v2/preview-harness.md, a closed bring-up doc forraw preview-next.Notes for reviewers
this branch; the preview work does not require them, so the two can land in
either order.
the promotion commit is mostly a module move (
raw/preview_next/→preview/)plus deletion of the V1 harness.