Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

[Question] Correct procedure for re-IPing a Pulsar cluster with ZooKeeper and BookKeeper #4694

Copy link
Copy link

Description

@Warre-Tielens
Issue body actions

Hello,

I’m looking for information on the correct way to re-IP an Apache Pulsar cluster using ZooKeeper and BookKeeper.

Before performing the re-IP, I wanted to make some configuration changes so that the BookieID uses <FQDN>:<port> instead of the default <IP>:<port>. To do this, I set useHostNameAsBookieID=true. However, after restarting the BookKeeper systemd service, the service fails to start and the following error appears in the logs:

2025-12-23T10:22:39,564 - ERROR [main:Main@218] - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [5
bookieHost: “<hostname>.dummy.com:3181"
journalDir: “/data/bookkeeper/journal01"
ledgerDirs: "1\t/data/bookkeeper/ledger01"
instanceId: "9a36eaf5-564e-4b5d-9636-24b082125688"
] is not matching with [5
bookieHost: “10.0.0.1:3181"
journalDir: "/data/bookkeeper/journal01"
ledgerDirs: "1\t/data/bookkeeper/ledger01"
instanceId: "9a36eaf5-564e-4b5d-9636-24b082125688"
]

I tried the following steps to resolve the issue:

  • Stopping the Bookkeeper systemd service
  • Moving the Cookie/VERSION files from the journal01 and ledger01 directories to /tmp
  • Deleting the old cookie using:
    • bin/bookkeeper shell cookie_delete 10.0.0.1:3181
  • Running updateBookieInLedger with the old srcBookie and new destBookie
    • bin/bookkeeper shell updateBookieInLedger --srcBookie ’10.0.0.1:3181' --destBookie ‘<hostname>.dummy.com:3181' --updatespersec 5
  • Generating a new cookie and coping the created file to the VERSION files for the journal01 and ledger01 directories.
    • bin/bookkeeper shell cookie_generate -j /data/bookkeeper/journal01 -l /data/bookkeeper/ledger01 -o /tmp/test-cookie <hostname>.dummy.com:3181

None of these steps unfortunately seemed to work. The only way I was able to recover was by clearing the entire journal and ledger directories and letting the node re-sync from the other bookies.

So my main questions are:

  • What is the correct/recommended approach to re-IP Bookkeeper nodes while preserving existing ledger data?
  • Are there any additional steps required when changing useHostNameAsBookieID on an existing cluster like clearing or updating the Cookie?

Some additional context about the environment:
Pulsar version: 3.0.5
Bookkeeper version: 4.16.5
Zookeeper version: 3.8.4
openJDK version: 17.0.17
Running on RHEL8 VM's

Thanks in advance!

Reactions are currently unavailable

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.