You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the runtime-next materialize leader startup, when a remote-authoritative
connector checkpoint is recovered ahead of the local committed frontier (its
embedded close clock matches the recovered hinted_close), the leader promotes
the hinted transaction to committed in memory only: committed_close and committed_frontier are advanced but never reconciled to RocksDB. A later crash
can then leave persisted state that matches neither the connector's authoritative
checkpoint nor the recovered hint, and startup bail!s — wedging the shard.
Impact
High: stuck shard / crash loop. Once wedged, the leader bail!s during startup
before running any transaction, so the connector never receives another commit and
its checkpoint stays put — the shard can't recover on its own. Requires a
remote-authoritative connector (one whose checkpoint the runtime treats as
authoritative) plus the crash sequence below.
The ahead-remote-authoritative arm (clock == Some(hinted_close) → committed_close = hinted_close; committed_frontier = committed_frontier.reduce(hinted_frontier))
does not set committed_frontier_rebuilt.
The startup reconcile Persist fires only on committed_frontier_rebuilt || delete_legacy_checkpoint, and even then never
writes committed_close_clock.
So the promotion stays in memory; RocksDB keeps the stale committed_close.
On the next recovery, frontier_mapping::extract_committed_close(&connector_checkpoint)
matching neither the recovered committed_close nor hinted_close hits the anyhow::bail!("...doesn't match Recover's committed_close ... or hinted_close ...") arm.
Reproduction (two crashes)
Let C0 = last durable committed close, H1 = the hinted txn the connector commits,
H2 = a later hint.
A txn hint-Persists at H1 but crashes before its commit Persist →
RocksDB {committed_close: C0, hinted_close: H1}.
Summary
In the runtime-next materialize leader startup, when a remote-authoritative
connector checkpoint is recovered ahead of the local committed frontier (its
embedded close clock matches the recovered
hinted_close), the leader promotesthe hinted transaction to committed in memory only:
committed_closeandcommitted_frontierare advanced but never reconciled to RocksDB. A later crashcan then leave persisted state that matches neither the connector's authoritative
checkpoint nor the recovered hint, and startup
bail!s — wedging the shard.Impact
High: stuck shard / crash loop. Once wedged, the leader
bail!s during startupbefore running any transaction, so the connector never receives another commit and
its checkpoint stays put — the shard can't recover on its own. Requires a
remote-authoritative connector (one whose checkpoint the runtime treats as
authoritative) plus the crash sequence below.
Where
crates/runtime-next/src/leader/materialize/startup.rsclock == Some(hinted_close)→committed_close = hinted_close; committed_frontier = committed_frontier.reduce(hinted_frontier))does not set
committed_frontier_rebuilt.committed_frontier_rebuilt || delete_legacy_checkpoint, and even then neverwrites
committed_close_clock.committed_close.frontier_mapping::extract_committed_close(&connector_checkpoint)matching neither the recovered
committed_closenorhinted_closehits theanyhow::bail!("...doesn't match Recover's committed_close ... or hinted_close ...")arm.Reproduction (two crashes)
Let C0 = last durable committed close, H1 = the hinted txn the connector commits,
H2 = a later hint.
RocksDB
{committed_close: C0, hinted_close: H1}.hinted_close→promote in memory (
committed_close = H1); RocksDB untouched.RocksDB
{committed_close: C0, hinted_close: H2}.RocksDB
{committed_close: C0, hinted_close: H2}.{committed_close: C0, hinted_close: H2}, connector reportsH1 → matches neither →
bail!→ shard wedged.