When running OpenXPKI as a cluster, improve the handling of divergent configuration versions.
Cluster members are expected to have the same configuration version. If they do, they are be eligible to process incoming requests.
Introduce additional logic to process joining an already running cluster:
When an OpenXPKI instance starts, it checks for the current cluster state. If there is no other cluster node active, it assumes an active role and serves requests.
If the node detects that there is already at least one other node that is already running in active state, it compares its own configuration version with the version of the active node.
If the versions match, the node joins the cluster and serves requests.
If the versions mismatch, the node produces a fatal error with a meaningful error message during the startup of the daemon process and aborts startup. This prevents multiple active nodes with different configuration versions.
When performing a seamless/rolling upgrade of a running cluster, this is the intended situation: the new node typically has the new, desired configuration. However, this new configuration will be rejected by the cluster.
In order to facilitate a seamless upgrade of the configuration, introduce a --force-accept-config-version (or similar) which causes the existing nodes to shut down their own operation administratively, either by exposing a failure on the /health endpoint or better by forcing a server shutdown of the other, running nodes once the node with the new configuration has signaled that it is ready to assume operation (note that this may also require activation of key material).
This results in the new node acting as sole active cluster member, and all other nodes dropping out of cluster service, allowing the operator to update the configuration on the other nodes and rejoin the cluster.
When running OpenXPKI as a cluster, improve the handling of divergent configuration versions.
Cluster members are expected to have the same configuration version. If they do, they are be eligible to process incoming requests.
Introduce additional logic to process joining an already running cluster:
When an OpenXPKI instance starts, it checks for the current cluster state. If there is no other cluster node active, it assumes an active role and serves requests.
If the node detects that there is already at least one other node that is already running in active state, it compares its own configuration version with the version of the active node.
If the versions match, the node joins the cluster and serves requests.
If the versions mismatch, the node produces a fatal error with a meaningful error message during the startup of the daemon process and aborts startup. This prevents multiple active nodes with different configuration versions.
When performing a seamless/rolling upgrade of a running cluster, this is the intended situation: the new node typically has the new, desired configuration. However, this new configuration will be rejected by the cluster.
In order to facilitate a seamless upgrade of the configuration, introduce a --force-accept-config-version (or similar) which causes the existing nodes to shut down their own operation administratively, either by exposing a failure on the /health endpoint or better by forcing a server shutdown of the other, running nodes once the node with the new configuration has signaled that it is ready to assume operation (note that this may also require activation of key material).
This results in the new node acting as sole active cluster member, and all other nodes dropping out of cluster service, allowing the operator to update the configuration on the other nodes and rejoin the cluster.