Redis Operator creates/configures/manages redis-failovers atop Kubernetes.

• Requirements
• Operator deployment on Kubernetes
  • Using the Helm chart
  • Using kubectl
  • Using kustomize
• Usage
  • Reducing update churn (--enable-hash)
  • Reconcile interval (--resync-period)
  • Protect the master from cluster-autoscaler eviction
  • Sentinel update strategy and PodDisruptionBudget
  • Persistence
  • NodeAffinity and Tolerations
• Topology Spread Contraints
  • Custom configurations
  • Custom shutdown script
  • Custom SecurityContext
  • Custom containerSecurityContext at container level
  • Custom command
  • Custom environment variables
  • Custom Priority Class
  • Custom Service Account
  • Custom Pod Annotations
  • Custom Service Annotations
  • Pausing reconciliation
  • Control of label propagation
  • ExtraVolumes and ExtraVolumeMounts
• Connection to the created Redis Failovers
  • Enabling redis auth
• example config
  • Bootstrapping from pre-existing Redis Instance(s)
  • Default versions
• Cleanup
  • Operator and CRD
  • Single Redis Failover
• Docker Images
  • Redis Operator
• Documentation

Kubernetes version: 1.21 or higher Redis version: 6 or higher

Redis operator is being tested against kubernetes 1.25 1.26 1.27 and redis 6 All dependencies have been vendored, so there's no need to any additional download.

In order to create Redis failovers inside a Kubernetes cluster, the operator has to be deployed. It can be done with plain old deployment, using Kustomize or with the provided Helm chart.

From the root folder of the project, execute the following:

helm repo add redis-operator https://spotahome.github.io/redis-operator
helm repo update
helm install redis-operator redis-operator/redis-operator

Helm chart only manage the creation of CRD in the first install. In order to update the CRD you will need to apply directly.

REDIS_OPERATOR_VERSION=v1.3.0
kubectl replace -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/manifests/databases.spotahome.com_redisfailovers.yaml

helm upgrade redis-operator redis-operator/redis-operator

To create the operator, you can directly create it with kubectl:

REDIS_OPERATOR_VERSION=v1.3.0
kubectl create -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/manifests/databases.spotahome.com_redisfailovers.yaml
kubectl apply -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/example/operator/all-redis-operator-resources.yaml

This will create a deployment named redisoperator.

The kustomize setup included in this repo is highly customizable using components, but it also comes with a few presets (in the form of overlays) supporting the most common use cases.

To install the operator with default settings and every necessary resource (including RBAC, service account, default resource limits, etc), install the default overlay:

kustomize build github.com/spotahome/redis-operator/manifests/kustomize/overlays/default

If you would like to customize RBAC or the service account used, you can install the minimal overlay.

Finally, you can install the full overlay if you want everything this operator has to offer, including Prometheus ServiceMonitor resources.

It's always a good practice to pin the version of the operator in your configuration to make sure you are not surprised by changes on the latest development branch:

kustomize build github.com/spotahome/redis-operator/manifests/kustomize/overlays/default?ref=v1.2.4

You can easily create your own config by creating a kustomization.yaml file (for example to apply custom resource limits, to add custom labels or to customize the namespace):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: redis-operator

commonLabels:
    foo: bar

resources:
  - github.com/spotahome/redis-operator/manifests/kustomize/overlays/full

Take a look at the manifests inside manifests/kustomize for more details.

Once the operator is deployed inside a Kubernetes cluster, a new API will be accesible, so you'll be able to create, update and delete redisfailovers.

In order to deploy a new redis-failover a specification has to be created:

REDIS_OPERATOR_VERSION=v1.2.4
kubectl create -f https://raw.githubusercontent.com/spotahome/redis-operator/${REDIS_OPERATOR_VERSION}/example/redisfailover/basic.yaml

This redis-failover will be managed by the operator, resulting in the following elements created inside Kubernetes:

• rfr-<NAME>: Redis configmap
• rfr-<NAME>: Redis statefulset
• rfr-<NAME>: Redis service (if redis-exporter is enabled)
• rfs-<NAME>: Sentinel configmap
• rfs-<NAME>: Sentinel deployment
• rfs-<NAME>: Sentinel service

NOTE: NAME is the named provided when creating the RedisFailover. IMPORTANT: the name of the redis-failover to be created cannot be longer that 48 characters, due to prepend of redis/sentinel identification and statefulset limitation.

By default the operator rewrites every owned resource (statefulset, deployment, configmaps, services, PDBs, RBAC) on every reconcile, whether or not anything changed. On clusters with mutating admission webhooks (Kyverno, OPA/Gatekeeper) this fires the webhook on every loop and bumps metadata.generation continuously.

Starting the operator with --enable-hash makes it store a hash of each resource it applies in a databases.spotahome.com/resource-hash annotation and skip the update when the resource it would write matches that hash.

Trade-off: while enabled, a resource that is edited by hand is no longer corrected on the next sync, because the operator's desired object is unchanged and so its hash still matches. You are trading the operator's self-healing of manual drift for far fewer API writes. The flag is off by default, so upgrading changes nothing until you opt in.

Beyond reacting to change events, the operator re-reconciles every RedisFailover on a fixed interval so it can recover from missed events and correct drift. This defaults to 30s and can be tuned with --resync-period (any Go duration, e.g. --resync-period=1m). Larger values reduce API load at the cost of slower periodic healing; smaller values react faster but poll more often.

Setting redis.preventMasterEviction: true makes the operator annotate the current master pod with cluster-autoscaler.kubernetes.io/safe-to-evict: "false" (and mark slaves "true"), so the cluster-autoscaler will not drain the node running the master and trigger an avoidable failover. The annotation follows the master as it moves. Defaults to false (no annotation is managed).

The sentinel Deployment update strategy can be overridden via sentinel.strategy (e.g. to set rollingUpdate.maxSurge/maxUnavailable). This helps when required anti-affinity plus replicas == nodes would otherwise deadlock the default rolling update:

spec:
  sentinel:
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 0

The PodDisruptionBudget minAvailable for each component defaults to 2 (or 1 when redis replicas <= 2). Override it per component with redis.podDisruptionBudgetMinAvailable / sentinel.podDisruptionBudgetMinAvailable (an integer or percentage string such as "60%").

The operator has the ability of add persistence to Redis data. By default an emptyDir will be used, so the data is not saved.

In order to have persistence, a PersistentVolumeClaim usage is allowed. The full PVC definition has to be added to the Redis Failover Spec under the Storage section.

IMPORTANT: By default, the persistent volume claims will be deleted when the Redis Failover is. If this is not the expected usage, a keepAfterDeletion flag can be added under the storage section of Redis. An example is given.

You can use NodeAffinity and Tolerations to deploy Pods to isolated groups of Nodes. Examples are given for node affinity, pod anti affinity and tolerations.

You can use the topologySpreadContraints to ensure the pods of a type(redis or sentinel) are evenly distributed across zones/nodes. Examples are for using topology spread constraints. Further document on how topologySpreadConstraints work could be found here.

It is possible to configure both Redis and Sentinel. This is done with the customConfig option inside their spec. It is a list of configurations and their values. Example are given in the custom config example file.

In order to have the ability of this configurations to be changed "on the fly", without the need of reload the redis/sentinel processes, the operator will apply them with calls to the redises/sentinels, using config set or sentinel set mymaster respectively. Because of this, no changes on the configmaps will appear regarding this custom configurations and the entries of customConfig from Redis spec will not be written on redis.conf file. To verify the actual Redis configuration use redis-cli CONFIG GET *.

Important: in the Sentinel options, there are some "conversions" to be made:

• Configuration on the sentinel.conf: sentinel down-after-milliseconds mymaster 2000
• Configuration on the configOptions: down-after-milliseconds 2000

Important 2: do NOT change the options used for control the redis/sentinel such as port, bind, dir, etc.

By default, a custom shutdown file is given. This file makes redis to SAVE it's data, and in the case that redis is master, it'll call sentinel to ask for a failover.

This behavior is configurable, creating a configmap and indicating to use it. An example about how to use this option can be found on the shutdown example file.

Important: the configmap has to be in the same namespace. The configmap has to have a shutdown.sh data, containing the script.

By default Kubernetes will run containers as the user specified in the Dockerfile (or the root user if not specified), this is not always desirable. If you need the containers to run as a specific user (or provide any other PodSecurityContext options) then you can specify a custom securityContext in the redisfailover object. See the SecurityContext example file for an example. You can visit kubernetes documentation for detailed docs about security context

A custom securityContext is merged on top of the operator defaults: fields you set win, and any field you leave unset keeps its default (e.g. setting only runAsUser no longer clears fsGroup/runAsNonRoot).

By default Kubernetes will run containers with default docker capabilities for example, this is not always desirable. If you need the containers to run with specific capabilities or with read only root file system (or provide any other securityContext options) then you can specify a custom containerSecurityContext in the redisfailover object. See the ContainerSecurityContext example file for an example. Keys available under containerSecurityContext are detailed here

A custom containerSecurityContext is merged on top of the operator defaults: fields you set win, and any field you leave unset keeps its default (e.g. the dropped ALL capabilities and allowPrivilegeEscalation: false are retained unless you override them).

By default, redis and sentinel will be called with the basic command, giving the configuration file:

• Redis: redis-server /redis/redis.conf
• Sentinel: redis-server /redis/sentinel.conf --sentinel

If necessary, this command can be changed with the command option inside redis/sentinel spec. An example can be found in the custom command example file.

Extra environment variables can be injected into the redis and sentinel main containers via redis.env / sentinel.env (standard Kubernetes EnvVar entries). The operator's own variables (REDIS_ADDR, REDIS_PORT, REDIS_USER, REDIS_PASSWORD) always take precedence, so a user-supplied variable that reuses one of those names cannot override it.

In order to use a custom Kubernetes Priority Class for Redis and/or Sentinel pods, you can set the priorityClassName in the redis/sentinel spec, this attribute has no default and depends on the specific cluster configuration. Note: the operator doesn't create the referenced Priority Class resource.

In order to use a custom Kubernetes Service Account for Redis and/or Sentinel pods, you can set the serviceAccountName in the redis/sentinel spec, if not specified the default Service Account will be used. Note: the operator doesn't create the referenced Service Account resource.

By default, no pod annotations will be applied to Redis nor Sentinel pods.

In order to apply custom pod Annotations, you can provide the podAnnotations option inside redis/sentinel spec. An example can be found in the custom annotations example file.

By default, no service annotations will be applied to the Redis nor Sentinel services.

In order to apply custom service Annotations, you can provide the serviceAnnotations option inside redis/sentinel spec. An example can be found in the custom annotations example file.

Setting the skip-reconcile annotation to "true" on a RedisFailover makes the operator ignore that resource until the annotation is removed or set back to "false". This is meant for migrations and maintenance windows, where the operator restoring the declared sentinel and replica counts would fight the operation in progress.

apiVersion: databases.spotahome.com/v1
kind: RedisFailover
metadata:
  name: redisfailover
  annotations:
    skip-reconcile: "true"
spec:
  # ...

While the annotation is set the operator neither ensures the owned resources nor runs its check-and-heal pass for that failover, so nothing repairs a broken master or replica until reconciliation is resumed. Other RedisFailover resources are unaffected.

By default the operator does not propagate the labels on the CRD down to the resources that it creates. To opt in to propagating a subset of them, list matching regular expressions in the labelWhitelist option in the spec.

Propagating labels blindly can be problematic when the labels on the CRD are not fully under your own control (for example: being deployed by a gitops operator): a change to a label value can fail on immutable resources such as PodDisruptionBudgets, and an applyset.kubernetes.io/* label reaching the operator's own resources can make kubectl apply --prune/ApplySet delete them out from under the operator.

Note: Older releases propagated all CRD labels when the whitelist was empty. That default was reversed to avoid the pruning hazard above. If you relied on the old behaviour, add a catch-all such as .* to labelWhitelist.

Items in the array should be regular expressions, see here as an example of how they can be used and here for a syntax reference.

The whitelist can also be used as a form of blacklist by specifying a regular expression that will not match any label.

NOTE: The operator will always add the labels it requires for operation to resources. These are the following:

app.kubernetes.io/component
app.kubernetes.io/managed-by
app.kubernetes.io/name
app.kubernetes.io/part-of
redisfailovers.databases.spotahome.com/name

If the user choose to have extra volumes creates and mounted, he could use the extraVolumes and extraVolumeMounts, in spec.redis of the CRD. This allows users to mount the extra configurations, or secrets to be used. A typical use case for this might be

• Secrets that sidecars might use to backup of RDBs
• Extra users and their secrets and acls that could used the initContainers to create multiple users
• Extra Configurations that could merge on top the existing configurations
• To pass failover scripts for addition for additional operations

---
apiVersion: v1
kind: Secret
metadata:
  name: foo
  namespace: exm
type: Opaque
stringData:
  password: MWYyZDFlMmU2N2Rm
---
apiVersion: databases.spotahome.com/v1
kind: RedisFailover
metadata:
  name: foo
  namespace: exm
spec:
  sentinel:
    replicas: 3
    extraVolumes:
    - name: foo
      secret:
        secretName: foo
        optional: false
    extraVolumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  redis:
    replicas: 3
    extraVolumes:
    - name: foo
      secret:
        secretName: foo
        optional: false
    extraVolumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true

In order to connect to the redis-failover and use it, a Sentinel-ready library has to be used. This will connect through the Sentinel service to the Redis node working as a master. The connection parameters are the following:

url: rfs-<NAME>
port: 26379
master-name: mymaster

To enable auth create a secret with a password field:

echo -n "pass" > password
kubectl create secret generic redis-auth --from-file=password

## example config
apiVersion: databases.spotahome.com/v1
kind: RedisFailover
metadata:
  name: redisfailover
spec:
  sentinel:
    replicas: 3
  redis:
    replicas: 1
  auth:
    secretPath: redis-auth

You need to set secretPath as the secret name which is created before.

The password is read from that Secret and passed to redis-server via --requirepass/--masterauth sourced from an environment variable; it is not written into the redis ConfigMap, so it never appears in plaintext in a ConfigMap or in the pod spec.

If you are wanting to migrate off of a pre-existing Redis instance, you can provide a bootstrapNode to your RedisFailover resource spec.

This bootstrapNode can be configured as follows:

When a bootstrapNode is provided, the Operator will always set all of the defined Redis instances to replicate from the provided bootstrapNode host value. This allows for defining a RedisFailover that replicates from an existing Redis instance to ease cutover from one instance to another.

Note: Redis instance will always be configured with replica-priority 0. This means that these Redis instances can never be promoted to a master.

Depending on the configuration provided, the Operator will launch the RedisFailover in two bootstrapping states: without sentinels and with sentinels.

By default, if the RedisFailover resource defines a valid bootstrapNode, only the redis instances will be created. This allows for ease of bootstrapping from an existing RedisFailover instance without the Sentinels intermingling with each other.

When allowSentinels is provided, the Operator will also create the defined Sentinel resources. These sentinels will be configured to point to the provided bootstrapNode as their monitored master.

The image versions deployed by the operator can be found on the defaults file.

If you want to delete the operator from your Kubernetes cluster, the operator deployment should be deleted.

Also, the CRD has to be deleted. Deleting CRD automatically wil delete all redis failover custom resources and their managed resources:

kubectl delete crd redisfailovers.databases.spotahome.com

Thanks to Kubernetes' OwnerReference, all the objects created from a redis-failover will be deleted after the custom resource is.

kubectl delete redisfailover <NAME>

For the code documentation, you can lookup on the GoDoc.

Also, you can check more deeply information on the docs folder.