Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

fix(ui): resolve delegate identity conflicts by generation, not arrival order (#527) - #528

#528
Merged
sanity merged 8 commits into
mainfreenet/river:mainfrom
fix-527-identity-rollbackfreenet/river:fix-527-identity-rollbackCopy head branch name to clipboard
Jul 28, 2026
Merged

fix(ui): resolve delegate identity conflicts by generation, not arrival order (#527)#528
sanity merged 8 commits into
mainfreenet/river:mainfrom
fix-527-identity-rollbackfreenet/river:fix-527-identity-rollbackCopy head branch name to clipboard

Conversation

@sanity

@sanity sanity commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Problem

The 2026-07-27 re-key publish (V29 → V30, for the DM retention cap in #524) forced every user through legacy delegate migration. A long-time user reported rooms coming back under identities from months earlier, on two of three nodes (#527).

fire_legacy_migration_request probes all 26 legacy delegate generations at once and each one holding data hydrates independently, so their responses race. Three defects turned that race into a rollback:

  1. Identity was decided by arrival order. Rooms::merge kept whichever copy landed FIRST and silently skipped any later copy with a different self_sk. Probes are dispatched oldest-generation-first, so the oldest surviving copy usually answered first and won. Only users whose identity for a room ever changed are affected — which is why this hit a long-time user and not everyone.

  2. The signing key pushed to the delegate ignored the merge outcome. The per-room keys were read from loaded_rooms BEFORE the (deferred) merge, so a room whose incoming copy the merge REJECTED still had that rejected key pushed — and migrate_signing_key overwrites on mismatch. The delegate was left signing with an identity the UI had already discarded, invalidating every signature it produced for that room.

  3. The migration could be sealed before the fan-out finished. The seal ends legacy migration permanently: once set, a later session whose current delegate is still empty finds it and probes nothing. The loudest door was in freenet_synchronizer.rs, which sealed on any "delegate not found" API error — an error the fan-out itself provokes within milliseconds on any node missing an old delegate WASM. One absent generation spoke for the other twenty-five.

Approach

Give the fan-out the ordering it was missing. LEGACY_DELEGATES is generated in age order, so an entry's index is its generation: higher is newer, and the current delegate outranks all of them.

  • resolve_identity_conflict decides by source authority. A strictly newer generation wins; equal ranks keep local, preserving the Import Identity should allow overwriting an existing room identity (warn, don't refuse) #414 stale-load guard.
  • merge_from_source threads the rank through and folds the local state into the incoming copy before adopting it, so no messages are lost. The map is not mutated until the fold succeeds — ChatRoomStateV1::merge is fallible, and mutating first meant an error deleted the room outright.
  • A deliberate identity choice (import, invitation-accept) is recorded authoritative synchronously at both sites, so no load can undo it.
  • Signing-key migration is deferred to run after the merge and reads the merged ROOMS.
  • Seal doors: the two where the current delegate is authoritative stay synchronous (no fan-out is in flight, and deferring weakens the bug: old delegates overwriting the current active delegate #253 guard); the two where a fan-out can be in flight are quiescence-gated, as is the delegate-not-found door. One writer, pinned.

Scope — please read before closing #527

  • This protects nodes that have not yet hit the bug. It is NOT recovery for those that have. An already-rolled-back user's current delegate is populated, so the per-room load path marks migration done and legacy is never re-probed. Shipping this changes nothing for them; they must re-import their identity, after which the fix keeps it.
  • A wrong ranking decision is now deterministic and permanent rather than a coin flip. For an already-affected user, the wrong identity sits in what becomes the highest-ranked legacy generation, so it would win every conflict thereafter. That is the right trade — deterministic-correct for the overwhelming majority beats a coin flip for everyone — but it should be stated rather than discovered.

Testing

Behavioural tests at the merge layer plus source pins for the wiring. Every pin was mutation-verified rather than trusted green; two tests were found vacuous that way and rebuilt — one of them passed with the bug fully restored, because its fixture made the fold succeed.

Review

Four independent blind lenses (code-first, skeptical/race, testing, data-loss/migration-compat), two rounds. Three reviewers independently found the same P1 (the map mutated before a fallible merge). All findings fixed or explicitly justified; final verdict non-blocking from all four.

Deliberately not fixed here, filed instead: #531 (keep-local path discards the loser's room_state — pre-existing, and closing it is a behaviour change deserving its own review) and #532 (interrupted-migration recovery has no seal bypass — pre-existing).

Closes #527

[AI-assisted - Claude]

…al order (#527)

## Problem

The 2026-07-27 re-key publish (V29 → V30) forced every user through legacy
delegate migration. A long-time user reported that several rooms came back
under identities from months earlier, on two of three nodes (#527).

`fire_legacy_migration_request` probes all 29 legacy delegate generations at
once and each one that still holds data hydrates independently, so their
responses race. Three separate defects turned that race into a rollback:

1. **Identity was decided by arrival order.** `Rooms::merge` kept whichever
   copy landed FIRST and silently skipped any later copy with a different
   `self_sk`. Probes are dispatched oldest-generation-first, so the oldest
   surviving copy usually answered first and won. Only users whose identity
   for a room ever changed are affected, which is why this hit a long-time
   user and not everyone.

2. **The signing key pushed to the delegate ignored the merge outcome.** The
   per-room signing keys were extracted from `loaded_rooms` BEFORE the
   (deferred) merge ran, so a room whose incoming copy the merge REJECTED
   still had that rejected key pushed to the delegate — and
   `migrate_signing_key` overwrites on mismatch. The delegate was left signing
   with an identity the UI had already discarded, invalidating every signature
   it produced for that room.

3. **The migration sealed too early.** `mark_legacy_migration_done()` fired as
   soon as the FIRST generation finished its re-save, while newer generations
   could still be answering. If the tab closed in that window, the current
   delegate was left holding the older copy — non-empty, so no later session
   ever re-probes legacy. That is what made a transient rollback permanent.

## Approach

Give the fan-out the ordering it was missing. `LEGACY_DELEGATES` is generated
from `legacy_delegates.toml` in age order, so an entry's index is its
generation: higher is newer, and the current delegate outranks all of them.

- `resolve_identity_conflict(local_rank, incoming_rank)` decides conflicts by
  source authority. A strictly newer generation wins; equal ranks keep local,
  which preserves the #414 stale-in-flight-load guard.
- `Rooms::merge_from_source` threads that rank through, and folds the local
  state into the incoming copy before adopting it so no messages are lost.
  `Rooms::merge` stays as the single-source wrapper with unchanged semantics.
- A deliberate identity choice (import, invitation-accept) is recorded as
  authoritative and outranks every delegate generation, so no load can undo an
  import. Marked synchronously at the import site as well as in
  `migrate_signing_key`, closing the window between the two.
- A room in memory with no recorded provenance is treated as authoritative and
  never downgraded, so an in-session-created room can't be overwritten.
- Signing-key migration is deferred to run after the merge and reads the
  merged `ROOMS`, so only the winning identity reaches the delegate.
- The legacy seal is armed on load quiescence, reusing the existing
  `LOAD_IDLE_MS` debounce: a later generation answering cancels it and re-arms
  when that worker settles.

This does not add the persisted identity-generation counter from #420 — that
orders identities the user changes from here on, and is a persisted-state
change with its own migration considerations. It is the natural next layer:
the same comparison, keyed on a counter first and source rank as the
tiebreak. Existing data all predates the counter, so source rank is what
recovers the users already affected.

## Testing

Eight behavioural tests plus three pins. Verified non-vacuous by mutation:
reverting `resolve_identity_conflict` to the old keep-local-always behaviour
turns the #527 regression test red, and mutating either source pin fails it.

- `a_later_arriving_newer_generation_corrects_an_older_one_that_answered_first`
  — the direct #527 reproduction.
- `identity_resolution_is_independent_of_arrival_order` — same two copies in
  the opposite order reach the same result, so the race is removed, not moved.
- `an_authoritative_identity_is_not_overwritten_by_any_generation` — #414.
- `unknown_provenance_is_treated_as_authoritative_and_never_downgraded`.
- `adopting_a_newer_identity_keeps_the_older_copys_messages`.
- `single_source_merge_still_keeps_local_on_conflict` — wrapper unchanged.
- `resolve_identity_conflict_prefers_a_strictly_newer_source`,
  `record_identity_source_takes_the_highest_rank_seen` — decision tables.
- `legacy_delegate_ranks_run_oldest_to_newest_and_below_the_current_delegate`
  — pins the load-bearing assumption that the registry is age-ordered. If a
  future generator sorted or reversed it, the fix would silently invert and
  start preferring the oldest copy.
- `signing_key_migration_reads_the_merged_rooms_not_the_loaded_copy`,
  `legacy_resave_defers_the_seal_to_quiescence` — source pins for causes 2/3.

735 river-ui unit tests pass; wasm build clean; no new clippy warnings.

Closes #527

[AI-assisted - Claude]
sanity added 7 commits July 28, 2026 10:35
Adopting a newer generation's copy replaced RoomData wholesale, dropping
any invitation-carried room secrets only the older copy held. These are
DECRYPTED per-version secrets — identity-independent recovery state — so
losing one leaves a private-room member unable to decrypt messages sealed
under that version.

Mirrors the save-side rule already pinned by
chat_delegate::reconcile_room_present_unions_invitation_secrets.
Asserting rank == index only tested position() against itself: had the
codegen ever emitted legacy_delegates.toml in a different order, every
index-based assertion would still have passed while the authority order
silently inverted — making the #527 fix prefer the OLDEST copy, the exact
bug it exists to prevent.

Anchors to V1 (2026-01-15) and V29 (2026-07-27), whose relative age is
known independently of array position, and asserts both are still
registry entries so the comparison can't pass by falling through to the
current-delegate rank.
P1: the delegate-rank ordering test was tautological — rank IS position()
over the array, so "rank == index" holds for ANY ordering, and anchoring
on V1/V29 does not help either (a lexicographic sort by version string
leaves V1 first and V29 late while V3 outranks V29). Now asserts the
emitted (YYYY-MM-DD) comments are non-decreasing, which array position
cannot influence, with a vacuity guard on the parse.

P1: nothing pinned the production WIRING. Swapping the shared registry
for a fresh map, or the threaded rank for a constant, reverts the whole
fix with every behavioural test still green — they all supply their own
ranks map. Pinned both, plus the per-generation rank at each call site.

P2: the seal pin asserted only that the re-save CALLS the request; the
debounce itself was unguarded. Pinned that the request never seals and
that the write happens only behind the idle gate, gate before write.
(That pin initially used the standard production split, which is wrong
for this file — its "mod tests" is mid-file, so the split hid both
functions under test. Uses whole-source rfind, as the file's other pins
do.)

P2: adopt replaces RoomData wholesale, so a new field is silently
dropped — exactly how the invitation_secrets loss got in. Added an
exhaustive destructure so adding a field breaks the build and forces a
decision, with each existing field's disposition recorded.

P2/P3: pinned the skip-on-absent "?" that stops a rejected identity
reaching the delegate; matched the real migrate_signing_key CALL rather
than a needle that also hit a comment; made the signature needle
rustfmt-proof; added a three-generation ordering test; restored a doc
comment that had landed on the wrong test.
Addresses the four-lens review of #528.

BLOCKER (found independently by three reviewers). The adopt path did
`map.remove()` BEFORE a fallible `room_state.merge(...)?`, so any merge
error returned with the room already gone: it vanished from the UI, every
room later in the iteration was dropped from that pass, and if that pass
was the one that re-saved to the current delegate the room was never
written there — permanently stranded, since legacy is then never probed
again. `ChatRoomStateV1::merge` genuinely errs here ("Private message
references unknown secret version N", "Cannot send public messages in
private room", unresolvable invite chains), and folding an older
generation into a newer one is exactly the shape that triggers it. The
map is now untouched until the fold succeeds, pinned by
`a_failing_fold_leaves_the_existing_room_untouched`.

SEAL DOORS. The seal ends legacy migration for good: once set, a later
session whose current delegate is still empty finds it and probes
nothing. Deferring only the re-save's seal left three inline doors open,
and the loudest was in freenet_synchronizer.rs — it seals on any
"delegate not found" API error, which the ~26-generation fan-out itself
provokes within milliseconds on any node where some legacy WASM was
never installed. One absent generation was speaking for the other
twenty-five, and if no generation had answered yet, nothing was ever
probed again. That is a plausible mechanism for the reporter's second
symptom ("found 1 room out of 8, then nothing no matter how long I
waited"). All four doors now route through the quiescence gate, leaving
one writer, pinned by `the_seal_has_exactly_one_writer`.

INVITATION-ACCEPT. Marks the identity authoritative synchronously, as
the identity-import path already did. Relying on the spawned
`migrate_signing_key` left a window where ROOMS held the new identity
while the registry still held a stale rank from an earlier load of the
same room, so a higher-ranked generation could adopt over a just-accepted
identity. The two rejoin paths now agree.

Also: carry `self_nickname` forward when the adopted copy has none (it is
a user preference, not identity-bound, and an imported-identity copy
carries None); correct a comment that still claimed "the local copy
always wins"; correct a doc claiming a diverging self_sk returns Err;
drop a false "several callers rely on it"; document that handing the
`merge` wrapper the shared registry would disable ranking; and record the
residual private-room caveat — adopting an identity that holds no
owner-signed blob for a secret version cannot read messages sealed under
it, which is inherent to the adopt, not a rebuild bug.

Deliberately NOT changed: the keep-local path still discards the incoming
copy's room_state. That asymmetry pre-dates this fix and closing it means
merging state across an identity conflict in the other direction too — a
behaviour change that deserves its own review rather than a rider on
this. The affected tests now say they certify identity convergence only.

742 river-ui unit tests pass; wasm build clean.
The fixture used an ordinary PUBLIC room, where the fold SUCCEEDS — so
the test passed even with the map mutated before the merge, which is
precisely the bug it exists to catch. Both error paths in
MessagesV1::apply_delta require the room to be private, so the newer
generation now flips the room to Private and the older copy's public
message trips 'Cannot send public messages in private room'.

Adds an explicit premise check on result.is_err(), so if contract
validation ever stops erroring here the test fails loudly instead of
going quietly vacuous again. Verified by mutation: restoring
remove-before-merge now turns it red.
… seal

Second review round.

Two reviewers independently showed that routing the two
current-delegate-is-authoritative doors through the debounce bought
nothing and cost something. Neither is a migration completion — there is
no fan-out in flight to wait for — and a deferred seal there is
cancellable by a reconnect, which weakens the #253 guard it exists to be.
Both are synchronous again, with the rationale recorded inline; the two
doors where a fan-out CAN be in flight stay gated, as does the
delegate-not-found door that caused the harm.

The seal is no longer written while a migration is in progress or after
one failed: the legacy re-save holds no load-worker guard, so quiescence
can be reached mid-flight, and fire_legacy_migration_request checks the
seal BEFORE the in-progress flag — so sealing there kills the #345
recovery across sessions. Not sealing just costs a harmless re-probe.

Rank provenance is now recorded BEFORE the fallible same-identity merge.
If that merge errored the rank was left under-stated, letting a later
OLDER generation out-rank it and adopt — the rollback this ordering
exists to prevent.

Tests: the single-writer pin now excises the mid-file test module and
counts the whole production file, rather than the ~20% ahead of the
module. My earlier note claimed the rest was covered by a companion pin;
it was not, and my stated reason for not counting properly was wrong —
the same brace trick the companion already uses works here. It now
encodes the real rule: an inline seal is allowed only at a door carrying
a written rationale, and there must be exactly two. Also adds the missing
test for the self_nickname carry-forward, which could have been deleted
with CI green — the same gap that let the invitation_secrets loss in.
@sanity

sanity commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Review summary

Full-tier review: four independent blind lenses, two full rounds plus a targeted delta check. Risk tier is Full because this touches delegate migration and per-room identity — a data-loss surface.

Lenses run: code-first (read the code before the description, flag intent/impl mismatch) · skeptical (adversarial race hunt, prior-fix regressions) · testing (vacuity, pin construction, coverage gaps) · data-loss / migration-compat (can this lose a room, a message, or an identity).

Blocking findings, all fixed

P1 — the adopt path mutated the map before a fallible merge. Found independently by three of the four lenses. map.remove() ran before room_state.merge(...)?, so any merge error returned with the room already gone: it vanished from the UI, the rest of that merge pass was abandoned, and if that pass was the one that re-saved to the current delegate, the room was never written there and legacy is never probed again — permanently stranded. ChatRoomStateV1::merge genuinely errs on this path. Fixed: the map is untouched until the fold succeeds.

P1 — nothing pinned the production wiring. Swapping the shared rank registry for a fresh map, or the threaded rank for a constant, reverted the whole fix with every behavioural test still green, because they all supply their own ranks map. Now pinned.

P1 — the delegate-rank ordering test was tautological. It asserted rank == index, which is an identity over position() and holds for any ordering — while its own doc claimed it guarded against reordering. Now asserts the emitted dates are non-decreasing.

Notable non-blocking outcomes

  • A seal door in freenet_synchronizer.rs — outside the original diff — sealed legacy migration on any "delegate not found" error, which the 26-generation fan-out itself provokes within milliseconds. One absent generation spoke for all the others. Now gated.
  • Two seal doors were initially over-gated; two lenses showed that deferring them bought nothing and weakened the bug: old delegates overwriting the current active delegate #253 guard. Reverted to synchronous with recorded rationale.
  • One reviewer retracted a finding with a precise reason (the scaffold macro re-clones state before each field, so an earlier field's delta is visible to later ones) — worth reading if you touch merge ordering.

Test discipline

Every pin was mutation-verified rather than trusted green. Two tests were found vacuous that way and rebuilt — one passed with the bug fully restored, because its fixture made the fold succeed rather than fail.

Deliberately not fixed

#531 (keep-local discards the loser's room_state — pre-existing, closing it is a behaviour change deserving its own review) and #532 (interrupted-migration recovery has no seal bypass — pre-existing, and annotated as having no traced producer).

[AI-assisted - Claude]

@sanity
sanity marked this pull request as ready for review July 28, 2026 16:50
@sanity
sanity merged commit 90e6dfa into main Jul 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lost IDs and rooms to River Update across two nodes.

1 participant

Morty Proxy This is a proxified and sanitized view of the page, visit original site.