This repository contains the GitOps manifests and operating notes for the Arencloud identity lab. The design connects Microsoft Active Directory, Evolveum midPoint, Red Hat Build of Keycloak, and a demo banking application named Balance.

The main goal is to prove this flow end to end:

midPoint assignment -> directory group -> RHBK realm role -> OIDC token -> application authorization

cl02 intentionally uses the same RHBK issuer hostname as cl03. That keeps application configuration stable during failover. While cl02 is passive, validate it with curl --resolve instead of publishing DNS to it.

clusters/
  cl03/
    applications/       Argo CD Applications for active site
    bootstrap/dns/      DNS forwarding for AD lookup
    operators/rhbk/     RHBK Operator subscription
    apps/rhbk/          Active RHBK runtime manifests
    apps/balance/       Balance application manifests
  cl02/
    applications/       Argo CD Applications for passive site
    bootstrap/dns/      DNS forwarding for AD lookup
    operators/rhbk/     RHBK Operator subscription
    apps/rhbk/          Passive RHBK runtime manifests
balance/                Quarkus OIDC demo banking application
balance-saml/           Quarkus SAML demo banking application
docs/                   Architecture, deployment, and operations notes

Run these from a shell logged into the target cluster.

Active site:

oc apply -k clusters/cl03/applications

Passive site:

oc apply -k clusters/cl02/applications

Useful status checks:

oc get application -n openshift-gitops
oc get pods -n rhbk
oc get externalsecret,certificate,keycloak,keycloakrealmimport,gateway,httproute -n rhbk
oc get gateway rhbk -n rhbk

Each cluster needs these capabilities before the RHBK application can converge:

Detailed setup for the non-OpenShift identity components is kept in dedicated documents:

At a high level, midPoint owns managed access:

Create/update user in midPoint
  -> assign Balance role
  -> midPoint updates AD and 389 DS group membership
  -> RHBK reads the selected LDAP federation source over LDAPS
  -> Balance receives roles in OIDC token

The arencloud realm has two LDAP user federation providers:

Both directories can contain the same usernames because midPoint provisions the same managed identities and Balance group memberships to both targets. Keycloak/RHBK does not merge duplicate federated users. A local imported user has exactly one Federation link, so the first provider that imports a username owns that local Keycloak user record.

Operationally:

• Keeping both providers enabled is allowed, but duplicate usernames must be managed deliberately.
• Provider priority decides lookup order for users that are not already imported.
• Existing imported users do not automatically move from AD to 389 DS, or from 389 DS back to AD.
• To switch ownership, remove imported users for the current provider, then sync the target provider.
• Removing imported users only clears Keycloak's local imported cache. It does not delete users from AD or 389 DS.

Switch a user set from AD to 389 DS:

RHBK Admin Console
  -> User federation
  -> arencloud-ad
  -> Action: Remove imported users
  -> arencloud-389ds
  -> Action: Sync all users

Switch a user set from 389 DS back to AD:

RHBK Admin Console
  -> User federation
  -> arencloud-389ds
  -> Action: Remove imported users
  -> arencloud-ad
  -> ensure Enabled is on
  -> Action: Sync all users

After switching, validate a user in Users and check:

Enabled: true
Federation link: arencloud-ad or arencloud-389ds
Groups: expected Balance groups

Secrets are not stored in Git. They are read from Vault by External Secrets.

The ExternalSecret remote keys are relative to the Vault mount configured in ClusterSecretStore/vault.

The balance/ directory contains the Quarkus application used to validate the identity and authorization flow.

cl03 GitOps deployment:

Balance authorizes users with RHBK realm roles:

The realm also includes a SAML client for testing SAML service-provider integration:

Use the RHBK realm SAML descriptor as the IdP metadata source:

https://sso.arencloud.com/realms/arencloud/protocol/saml/descriptor

The SAML client is used by the second demo application in balance-saml.

See balance/README.md for local development, API routes, and browser test flows.

The balance-saml/ directory contains the second banking demo application. It tests RHBK as a SAML Identity Provider with a Quarkus service-provider application.

cl03 GitOps deployment:

Passive cl02 GitOps deployment:

See balance-saml/README.md for local build, SAML endpoints, and OpenShift runtime notes.

Active site DNS validation:

dig +short sso.arencloud.com
curl -vk https://sso.arencloud.com/realms/arencloud/.well-known/openid-configuration
curl -vk https://sso.arencloud.com/realms/arencloud/protocol/saml/descriptor
curl -vk https://balance.arencloud.com/
curl -vk https://balance-saml.arencloud.com/

Passive cl02 validation without moving DNS:

CL02_GW_IP=$(oc get gateway rhbk -n rhbk -o jsonpath='{.status.addresses[0].value}')

curl -vk --resolve sso.arencloud.com:443:${CL02_GW_IP} \
  https://sso.arencloud.com/realms/arencloud/.well-known/openid-configuration

Gateway and LoadBalancer checks:

oc get gateway -n rhbk rhbk
oc get svc -A --field-selector spec.type=LoadBalancer
oc get ipaddresspool,l2advertisement -n metallb-system

cl02 is prepared as a passive site, not an active/active Keycloak cluster.

Before failover, confirm:

• cl02 RHBK is healthy through curl --resolve.
• The balance client secret and realm signing key strategy match the chosen DR model.
• Cloudflare or ExternalDNS ownership is changed deliberately so only the intended site publishes sso.arencloud.com.
• Users may need to sign in again in the warm-standby model.

The detailed runbook is in cl02 active/passive GitOps deployment.

• Do not commit secrets, tokens, exported client secrets, or generated admin passwords.
• Keep platform bootstrap and application manifests separate.
• Treat sso.arencloud.com as the stable issuer; change DNS publication, not application issuer configuration, during failover.
• Validate passive cl02 with curl --resolve until failover is approved.
• Keep generated RHBK realm secrets and signing keys under deliberate operational control before claiming seamless failover.