Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

@kjnilsson addressed the feedback in 29583b2 — transfer_leadership_of_stream/2 now skips the restart_stream/2 call when no online, non-maintenance replica is available among the transfer candidates, and logs at info level instead. Ready for another look when you have a moment.

bcdb0f8fba did not fix it — the sync check on rabbit_stream_coordinator:members/1 only proves the replicas are running as members, not that their stopped-tail matches the writer's. Both parallel-ct-set-5 runs failed on that commit with the same log line: Stream ... new leader is on node rmq-ct-stream_leadership_transfer-1@localhost, epoch 3.

Root cause is in rabbit_stream_coordinator:select_leader/2 (machine version 4+): preferred_leader_node is only honoured when the preferred node is tied with the current writer on the stopped tail. Right after restart_stream advances an epoch on the local node, the writer's tail is {Epoch, 0} while replicas still report empty, so the writer ends up alone in Potential and re-elects itself regardless of the hint.

Fixed in beb58de9cd at the drain site: retry restart_stream/2 up to three times with a 500 ms backoff when the returned leader is the local node. Between attempts the members run under the new epoch long enough for the replicas to catch up, so the next call finds their tails tied with the writer and the preferred_leader_node hint wins. The test change from bcdb0f8fba is reverted.

I'm testing this PR on Kubernetes. I have a 3-node cluster with stream-perf-test -sc 100 -x 100 -y 0 -r 1000 running.
When I perform kubectl rollout restart sts rmq-server, and rmq-server-2 nodes is being drained, rmq-server-0 and rmq-server-1 log errors like this:

** Generic server <0.15003.0> terminating
** Last message in was {continue,
                        #{name => "__stream-036_1783429184925345148",
                          nodes =>
                           ['rabbit@rmq-server-2.rmq-nodes.foo',
                            'rabbit@rmq-server-0.rmq-nodes.foo',
                            'rabbit@rmq-server-1.rmq-nodes.foo'],
                          reference =>
                           {resource,<<"/">>,queue,<<"stream-036">>},
                          features => #{committed_offset_calculate => true},
                          leader_pid => <31794.14060.0>,
                          leader_node => 'rabbit@rmq-server-2.rmq-nodes.foo',
                          replica_nodes =>
                           ['rabbit@rmq-server-0.rmq-nodes.foo',
                            'rabbit@rmq-server-1.rmq-nodes.foo'],
                          epoch => 3,
                          event_formatter =>
                           {rabbit_stream_queue,format_osiris_event,
                            [{resource,<<"/">>,queue,<<"stream-036">>}]},
                          max_segment_size_bytes => 500000000,
                          retention => [{max_bytes,20000000000}],
                          filter_size => 16}}
** When Server state == undefined
** Reason for termination ==
** {badrpc,{'EXIT',killed}}

  crasher:
    initial call: osiris_replica:init/1
    pid: <0.15003.0>
    registered_name: []
    exception exit: {badrpc,{'EXIT',killed}}
      in function  gen_server:handle_common_reply/5 (gen_server.erl:2562)
    ancestors: [osiris_server_sup,osiris_sup,<0.131.0>]

Problems can be seen with a simple make start-cluster as well:

make start-cluster
stream-perf-test  -sc 100 -x 100 -y 0 -r 1000
make restart-cluster

I'm seeing:

[warning] <0.2653.0> Coordinator timeout on server 'rabbit-3@FH7Q7R4JQ1' when processing command restart_stream
[warning] <0.2653.0> Failed to transfer leadership of stream queue 'stream-093' in vhost '/': {error, coordinator_unavailable}

and:

[warning] <0.1516.0> Error/Timeout when retrieving broker ''rabbit-2@FH7Q7R4JQ1'' metadata: {error, {erpc, timeout}}

After a few minutes, it's still trying to drain the first node. As soon as I stop stream-perf-test, the restart can make progress and succeeds pretty quickly.

@mkuratczyk thanks for running this on a real cluster — those two reports pointed at a genuine problem with the drain loop under load. Pushed 571f7ff.

On the make start-cluster + stream-perf-test case (Coordinator timeout on server rabbit-3…, {error, coordinator_unavailable} on stream-093)

Root cause was on my side: the drain loop was firing restart_stream/2 serially for every local leader with up to 3 retries. Each command can wait up to stream_cmd_timeout (30 s) per raft member before falling through — so a single overloaded coordinator turns into minutes of wasted wall-clock while more commands pile up behind. That's why the drain never made progress until stream-perf-test stopped.

transfer_leadership_of_local_stream_leaders/1 now:

• caps total wall-clock at 30 s (remaining streams fall back to shutdown-time re-election);
• bails out on the first {error, coordinator_unavailable} or {timeout, _} from restart_stream/2 — hammering an overloaded coordinator only prolongs the outage;
• drops retries from 3 to 2;
• inserts a 25 ms breather between successful transfers so the coordinator command queue can drain;
• logs a transferred=N, skipped=N, remaining=N summary so it's clear from the logs what actually happened.

On the K8s {badrpc,{'EXIT',killed}} from osiris_replica on the surviving nodes

That's restart_stream/2 semantics: the coordinator sets target = stopped for every member (writer + replicas), so replicas on the other nodes are killed and restarted alongside the writer. osiris_server_sup brings them back, but the transient stop-restart shows up as ugly crash reports on the surviving nodes. This is inherent to the current coordinator API — I don't see a lighter "just move the writer" primitive in rabbit_stream_coordinator. Happy to explore adding one (or capping the number of parallel restarts) if you think that's the right direction, but it feels like a separate change.

Ready for another test cycle when you have a moment. Title updated back — the DO NOT MERGE tag was left on so the maintainer flip is intentional if you want to keep it there.

cc @kjnilsson