Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

runtime-v2: ahead-remote-authoritative recovery promotes committed_close in memory only #3176

Copy link
Copy link

Description

@williamhbaker
Issue body actions

Summary

In the runtime-next materialize leader startup, when a remote-authoritative
connector checkpoint is recovered ahead of the local committed frontier (its
embedded close clock matches the recovered hinted_close), the leader promotes
the hinted transaction to committed in memory only: committed_close and
committed_frontier are advanced but never reconciled to RocksDB. A later crash
can then leave persisted state that matches neither the connector's authoritative
checkpoint nor the recovered hint, and startup bail!s — wedging the shard.

Impact

High: stuck shard / crash loop. Once wedged, the leader bail!s during startup
before running any transaction, so the connector never receives another commit and
its checkpoint stays put — the shard can't recover on its own. Requires a
remote-authoritative connector (one whose checkpoint the runtime treats as
authoritative) plus the crash sequence below.

Where

crates/runtime-next/src/leader/materialize/startup.rs

  • The ahead-remote-authoritative arm (clock == Some(hinted_close)
    committed_close = hinted_close; committed_frontier = committed_frontier.reduce(hinted_frontier))
    does not set committed_frontier_rebuilt.
  • The startup reconcile Persist fires only on
    committed_frontier_rebuilt || delete_legacy_checkpoint, and even then never
    writes committed_close_clock.
  • So the promotion stays in memory; RocksDB keeps the stale committed_close.
  • On the next recovery, frontier_mapping::extract_committed_close(&connector_checkpoint)
    matching neither the recovered committed_close nor hinted_close hits the
    anyhow::bail!("...doesn't match Recover's committed_close ... or hinted_close ...") arm.

Reproduction (two crashes)

Let C0 = last durable committed close, H1 = the hinted txn the connector commits,
H2 = a later hint.

  1. A txn hint-Persists at H1 but crashes before its commit Persist →
    RocksDB {committed_close: C0, hinted_close: H1}.
  2. Recovery: connector checkpoint reports H1 == recovered hinted_close
    promote in memory (committed_close = H1); RocksDB untouched.
  3. Leader runs; the first txn hint-Persists at H2 →
    RocksDB {committed_close: C0, hinted_close: H2}.
    RocksDB {committed_close: C0, hinted_close: H2}.
  4. Crash before the connector commits (its checkpoint still reports H1).
  5. Recovery: RocksDB {committed_close: C0, hinted_close: H2}, connector reports
    H1 → matches neither → bail! → shard wedged.
Reactions are currently unavailable

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

Morty Proxy This is a proxified and sanitized view of the page, visit original site.