Rotate Authentication of Memcached

Rotate Authentication is a feature of the KubeDB Ops-Manager that allows you to rotate a Memcached user’s authentication credentials using a MemcachedOpsRequest. There are two ways to perform this rotation.

  1. Operator Generated: The KubeDB operator automatically generates a random credential, updates the existing secret with the new credential, and does not provide the secret details directly to the user.
  2. User Defined: The user can create their own credentials by defining a Secret of type kubernetes.io/basic-auth containing the desired username and password, and then reference this Secret in the MemcachedOpsRequest.

Before You Begin

  • You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind.

  • Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps here. Make sure install with helm command including --set global.featureGates.Memcached=true to ensure Memcached CRD installation.

  • To configure TLS/SSL in Memcached, KubeDB uses cert-manager to issue certificates. So first you have to make sure that the cluster has cert-manager installed. To install cert-manager in your cluster following steps here.

  • You should be familiar with the following KubeDB concepts:

To keep everything isolated, we are going to use a separate namespace called demo throughout this tutorial.

$ kubectl create ns demo
namespace/demo created

Note: YAML files used in this tutorial are stored in docs/examples/memcached directory of kubedb/docs repository.

Create a Memcached Server

KubeDB implements a Memcached CRD to define the specification of a Memcached server. Below is the Memcached object created in this tutorial.

Note: If your KubeDB version is less or equal to v2024.6.4, You have to use v1alpha2 apiVersion.

apiVersion: kubedb.com/v1
kind: Memcached
metadata:
  name: memcd-quickstart
  namespace: demo
spec:
  replicas: 1
  version: "1.6.22"
  podTemplate:
    spec:
      containers:
        - name: memcached
          resources:
            limits:
              cpu: 500m
              memory: 128Mi
            requests:
              cpu: 250m
              memory: 64Mi
  deletionPolicy: DoNotTerminate
$ kubectl create -f https://github.com/kubedb/docs/raw/v2025.7.31/docs/examples/memcached/quickstart/demo-v1.yaml
memcached.kubedb.com/memcd-quickstart created

Now, wait until memcd-quickstart has status Ready. i.e,

$  kubectl get mc -n demo -w
NAME               VERSION   STATUS   AGE
memcd-quickstart   1.6.22    Ready    17h

Verify Authentication

The user can verify whether they are authorized by executing a query directly in the database. To do this, the user needs username and password in order to connect to the database. Below is an example showing how to retrieve the credentials from the Secret.

$ kubectl get memcached -n demo memcd-quickstart -ojson | jq .spec.authSecret.name
"memcd-quickstart-auth"
$ kubectl get secret -n demo memcd-quickstart-auth -o=jsonpath='{.data.authData}' | base64 -d
user:ikbkjbodeewenrgj

Here, usernameis user and password is ikbkjbodeewenrgj

Connect With Memcached Database Using Credentials

Here, we will connect to Memcached server from local-machine through port-forwarding. We will connect to memcd-quickstart-0 pod from local-machine using port-frowarding and it must be running in separate terminal.

$ kubectl port-forward -n demo memcd-quickstart-0 11211
Forwarding from 127.0.0.1:11211 -> 11211
Forwarding from [::1]:11211 -> 11211

Now, you can connect to this database using telnet.Connect to Memcached from local-machine through telnet.


~ $ telnet 127.0.0.1 11211
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

version
CLIENT_ERROR unauthenticated # that means still you can not enter in DB

# Save data Command:
set my_key 0 2592000 21
# Meaning:
# 0       => no flags
# 2592000 => TTL (Time-To-Live) in [s]
# 21       => credential size in bytes

user bvkwwqxekxouudbr  # username password 

# Output:
STORED

#now you can use DB
version
VERSION 1.6.22

# Exit
quit

If you can access the data table and run queries, it means the secrets are working correctly.

Create RotateAuth MemcachedOpsRequest

1. Using Operator Generated Credentials:

In order to rotate authentication to the Memcached using operator generated, we have to create a MemcachedOpsRequest CRO with RotateAuth type. Below is the YAML of the MemcachedOpsRequest CRO that we are going to create,

apiVersion: ops.kubedb.com/v1alpha1
kind: MemcachedOpsRequest
metadata:
  name: mcops-rotate-auth-generated
  namespace: demo
spec:
  type: RotateAuth
  databaseRef:
    name: memcd-quickstart
  timeout: 5m
  apply: IfReady

Here,

  • spec.databaseRef.name specifies that we are performing rotate authentication operation on memcd-quickstart cluster.
  • spec.type specifies that we are performing RotateAuth on Memcached.

Let’s create the MemcachedOpsRequest CR we have shown above,

 $ kubectl apply -f https://github.com/kubedb/docs/raw/{{ .version }}/docs/examples/memcached/rotate-auth/rotate-auth-generated.yaml
 Memcachedopsrequest.ops.kubedb.com/mcops-rotate-auth-generated created

Let’s wait for MemcachedOpsrequest to be Successful. Run the following command to watch MemcachedOpsrequest CRO

 $ kubectl get Memcachedopsrequest -n demo
 NAME                          TYPE         STATUS       AGE
 mcops-rotate-auth-generated   RotateAuth   Successful   7m47s

If we describe the MemcachedOpsRequest we will get an overview of the steps that were followed.

$ kubectl describe MemcachedopsRequest -n demo mcops-rotate-auth-generated
Name:         mcops-rotate-auth-generated
Namespace:    demo
Labels:       <none>
Annotations:  <none>
API Version:  ops.kubedb.com/v1alpha1
Kind:         MemcachedOpsRequest
Metadata:
  Creation Timestamp:  2025-08-21T10:08:37Z
  Generation:          1
  Resource Version:    147775
  UID:                 2466e827-fee0-4f58-89f5-2cd2265cf1a6
Spec:
  Apply:  IfReady
  Database Ref:
    Name:   memcd-quickstart
  Timeout:  5m
  Type:     RotateAuth
Status:
  Conditions:
    Last Transition Time:  2025-08-21T10:08:37Z
    Message:               Memcached Ops Request has started to rotate auth for Memcached
    Observed Generation:   1
    Reason:                RotateAuth
    Status:                True
    Type:                  RotateAuth
    Last Transition Time:  2025-08-21T10:08:40Z
    Message:               Successfully generated new Credentials
    Observed Generation:   1
    Reason:                UpdateCredential
    Status:                True
    Type:                  UpdateCredential
    Last Transition Time:  2025-08-21T10:08:40Z
    Message:               Successfully Updated PetSets for rotate auth type
    Observed Generation:   1
    Reason:                UpdatePetSets
    Status:                True
    Type:                  UpdatePetSets
    Last Transition Time:  2025-08-21T10:08:51Z
    Message:               Restarted pods after rotate auth
    Observed Generation:   1
    Reason:                RestartPods
    Status:                True
    Type:                  RestartPods
    Last Transition Time:  2025-08-21T10:08:46Z
    Message:               evict pod; ConditionStatus:True; PodName:memcd-quickstart-0
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--memcd-quickstart-0
    Last Transition Time:  2025-08-21T10:08:46Z
    Message:               is pod ready; ConditionStatus:False
    Observed Generation:   1
    Status:                False
    Type:                  IsPodReady
    Last Transition Time:  2025-08-21T10:08:51Z
    Message:               is pod ready; ConditionStatus:True; PodName:memcd-quickstart-0
    Observed Generation:   1
    Status:                True
    Type:                  IsPodReady--memcd-quickstart-0
    Last Transition Time:  2025-08-21T10:08:51Z
    Message:               Successfully Rotated Memcached Auth Secret
    Observed Generation:   1
    Reason:                Successful
    Status:                True
    Type:                  Successful
  Observed Generation:     1
  Phase:                   Successful
Events:
  Type     Reason                                                          Age   From                         Message
  ----     ------                                                          ----  ----                         -------
  Normal   PauseDatabase                                                   112s  KubeDB Ops-manager Operator  Pausing Memcached demo/memcd-quickstart
  Normal   VersionUpdate                                                   109s  KubeDB Ops-manager Operator  Updating PetSets
  Normal   VersionUpdate                                                   109s  KubeDB Ops-manager Operator  Successfully Updated PetSets
  Warning  evict pod; ConditionStatus:True; PodName:memcd-quickstart-0     103s  KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:memcd-quickstart-0
  Warning  is pod ready; ConditionStatus:False                             103s  KubeDB Ops-manager Operator  is pod ready; ConditionStatus:False
  Warning  is pod ready; ConditionStatus:True; PodName:memcd-quickstart-0  98s   KubeDB Ops-manager Operator  is pod ready; ConditionStatus:True; PodName:memcd-quickstart-0
  Normal   RestartPods                                                     98s   KubeDB Ops-manager Operator  Restarted pods after rotate auth
  Normal   ResumeDatabase                                                  98s   KubeDB Ops-manager Operator  Resuming Memcached demo/memcd-quickstart
  Normal   ResumeDatabase                                                  98s   KubeDB Ops-manager Operator  Successfully resumed Memcached demo/memcd-quickstart
  Normal   Successful                                                      98s   KubeDB Ops-manager Operator  Successfully Rotated Memcached Auth Secret for demo/memcd-quickstart

Verify Auth is rotated

$ kubectl get mc -n demo memcd-quickstart -ojson | jq .spec.authSecret.name
"memcd-quickstart-auth"
$ kubectl get secret -n demo memcd-quickstart-auth -o=jsonpath='{.data.authData}' | base64 -d
user:yjf3Oc;ZlSs.iMVO                    

Let’s verify whether the credential is working or not:

We will connect to memcd-quickstart-0 pod from local-machine using port-frowarding and it must be running in separate terminal.

$ kubectl port-forward -n demo memcd-quickstart-0 11211
Forwarding from 127.0.0.1:11211 -> 11211
Forwarding from [::1]:11211 -> 11211

Now, you can connect to this database using telnet.Connect to Memcached from local-machine through telnet.


$ telnet 127.0.0.1 11211
#Output
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

# Save data Command:
set my_key 0 2592000 21
# Meaning:
# 0       => no flags
# 2592000 => TTL (Time-To-Live) in [s]
# 21       => credential size in bytes
user yjf3Oc;ZlSs.iMVO  
#ouput
STORED
#command
version
#output
VERSION 1.6.22
#output
quit
Connection closed by foreign host.

Your credentials have been stored successfully, so everything’s working. Also, there will be two more new keys in the secret that stores the previous credentials. The key is authData.prev. You can find the secret and its data by running the following command:

$ kubectl get secret -n demo memcd-quickstart-auth -o go-template='{{ index .data "authData.prev" }}' | base64 -d
user:ikbkjbodeewenrgj

The above output shows that the password has been changed successfully. The previous username & password is stored for rollback purpose.

2. Using User Created Credentials

At first, we need to create a secret with kubernetes.io/basic-auth type using custom username and password. Below is the command to create a secret with kubernetes.io/basic-auth type, You can use the following yaml regarding changing the credentials

apiVersion: v1
data:
  authData: dXNlcjpwYXNzCg==
kind: Secret
metadata:
   name: mc-new-auth
   namespace: demo
type: Opaque

Here, The data.authdata field stores user credentials in Base64-encoded format as username:password. Now create a MemcachedOpsRequest with RotateAuth type. Below is the YAML of the MemcachedOpsRequest that we are going to create,

Let’s create the secret

kubectl apply -f https://github.com/kubedb/docs/raw/{{ .version }}/docs/examples/memcached/rotate-auth/secret.yaml
secret/mc-new-auth created
apiVersion: ops.kubedb.com/v1alpha1
kind: MemcachedOpsRequest
metadata:
  name: mcops-rotate-auth-user
  namespace: demo
spec:
  type: RotateAuth
  databaseRef:
    name: memcached-quickstart
  authentication:
    secretRef:
      name: mc-new-auth
  timeout: 5m
  apply: IfReady

Here,

  • spec.databaseRef.name specifies that we are performing rotate authentication operation on memcd-quickstartcluster.
  • spec.type specifies that we are performing RotateAuth on Memcached.
  • spec.authentication.secretRef.name specifies that we are using mc-new-auth as spec.authSecret.name for authentication.

Let’s create the MemcachedOpsRequest CR we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/{{ .version }}/docs/examples/memcached/rotate-auth/rotate-auth-user.yaml
Memcachedopsrequest.ops.kubedb.com/mcops-rotate-auth-user created

Let’s wait for MemcachedOpsRequest to be Successful. Run the following command to watch MemcachedOpsRequest CRO:

$ kubectl get Memcachedopsrequest -n demo
NAME                          TYPE         STATUS       AGE
mcops-rotate-auth-generated   RotateAuth   Successful   19h
mcops-rotate-auth-user        RotateAuth   Successful   7m44s

We can see from the above output that the MemcachedOpsRequest has succeeded. If we describe the MemcachedOpsRequest we will get an overview of the steps that were followed.

$ kubectl describe Memcachedopsrequest -n demo mcops-rotate-auth-user
Name:         mcops-rotate-auth-user
Namespace:    demo
Labels:       <none>
Annotations:  <none>
API Version:  ops.kubedb.com/v1alpha1
Kind:         MemcachedOpsRequest
Metadata:
  Creation Timestamp:  2025-08-21T11:39:01Z
  Generation:          1
  Resource Version:    159639
  UID:                 6d289952-fd05-4311-985e-9a7fa7845953
Spec:
  Apply:  IfReady
  Authentication:
    Secret Ref:
      Name:  mc-new-auth
  Database Ref:
    Name:   memcd-quickstart
  Timeout:  5m
  Type:     RotateAuth
Status:
  Conditions:
    Last Transition Time:  2025-08-21T11:39:01Z
    Message:               Memcached Ops Request has started to rotate auth for Memcached
    Observed Generation:   1
    Reason:                RotateAuth
    Status:                True
    Type:                  RotateAuth
    Last Transition Time:  2025-08-21T11:39:01Z
    Message:               Successfully referenced the user provided authSecret
    Observed Generation:   1
    Reason:                UpdateCredential
    Status:                True
    Type:                  UpdateCredential
    Last Transition Time:  2025-08-21T11:39:01Z
    Message:               Successfully Updated PetSets for rotate auth type
    Observed Generation:   1
    Reason:                UpdatePetSets
    Status:                True
    Type:                  UpdatePetSets
    Last Transition Time:  2025-08-21T11:39:11Z
    Message:               Restarted pods after rotate auth
    Observed Generation:   1
    Reason:                RestartPods
    Status:                True
    Type:                  RestartPods
    Last Transition Time:  2025-08-21T11:39:06Z
    Message:               evict pod; ConditionStatus:True; PodName:memcd-quickstart-0
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--memcd-quickstart-0
    Last Transition Time:  2025-08-21T11:39:06Z
    Message:               is pod ready; ConditionStatus:False
    Observed Generation:   1
    Status:                False
    Type:                  IsPodReady
    Last Transition Time:  2025-08-21T11:39:11Z
    Message:               is pod ready; ConditionStatus:True; PodName:memcd-quickstart-0
    Observed Generation:   1
    Status:                True
    Type:                  IsPodReady--memcd-quickstart-0
    Last Transition Time:  2025-08-21T11:39:11Z
    Message:               Successfully Rotated Memcached Auth Secret
    Observed Generation:   1
    Reason:                Successful
    Status:                True
    Type:                  Successful
  Observed Generation:     1
  Phase:                   Successful
Events:
  Type     Reason                                                          Age   From                         Message
  ----     ------                                                          ----  ----                         -------
  Normal   VersionUpdate                                                   13m   KubeDB Ops-manager Operator  Updating PetSets
  Normal   VersionUpdate                                                   13m   KubeDB Ops-manager Operator  Successfully Updated PetSets
  Warning  evict pod; ConditionStatus:True; PodName:memcd-quickstart-0     13m   KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:memcd-quickstart-0
  Warning  is pod ready; ConditionStatus:False                             13m   KubeDB Ops-manager Operator  is pod ready; ConditionStatus:False
  Warning  is pod ready; ConditionStatus:True; PodName:memcd-quickstart-0  13m   KubeDB Ops-manager Operator  is pod ready; ConditionStatus:True; PodName:memcd-quickstart-0
  Normal   RestartPods                                                     13m   KubeDB Ops-manager Operator  Restarted pods after rotate auth
  Normal   ResumeDatabase                                                  13m   KubeDB Ops-manager Operator  Resuming Memcached demo/memcd-quickstart
  Normal   ResumeDatabase                                                  13m   KubeDB Ops-manager Operator  Successfully resumed Memcached demo/memcd-quickstart
  Normal   Successful                                                      13m   KubeDB Ops-manager Operator  Successfully Rotated Memcached Auth Secret for demo/memcd-quickstart

Verify auth is rotate

$ kubectl get mc -n demo memcd-quickstart -ojson | jq .spec.authSecret.name
"mc-new-auth"
$  kubectl get secret -n demo mc-new-auth -o=jsonpath='{.data.authData}' | base64 -d
user:pass                                                                                   

Let’s verify whether the credential is working or not:

We will connect to memcd-quickstart-0 pod from local-machine using port-frowarding and it must be running in separate terminal.

$ kubectl port-forward -n demo memcd-quickstart-0 11211
Forwarding from 127.0.0.1:11211 -> 11211
Forwarding from [::1]:11211 -> 11211

Now, you can connect to this database using telnet.Connect to Memcached from local-machine through telnet.


$ telnet 127.0.0.1 11211
#Output
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

# Save data Command:
set my_key 0 2592000 9
# Meaning:
# 0       => no flags
# 2592000 => TTL (Time-To-Live) in [s]
# 9      => credential size in bytes
user pass
#ouput
STORED
#command
version
#output
VERSION 1.6.22
#output
quit
Connection closed by foreign host.

Your credentials have been stored successfully, so everything’s working.

Also, there will be two more new keys in the secret that stores the previous credentials. The keys are username.prev and password.prev. You can find the secret and its data by running the following command:

$ kubectl get secret -n demo mc-new-auth -o go-template='{{ index .data "authData.prev" }}' | base64 -d
user:yjf3Oc;ZlSs.iMVO
                  

The above output shows that the password has been changed successfully. The previous username & password is stored in the secret for rollback purpose.

Cleaning Up

To clean up the Kubernetes resources you can delete the CRD or namespace. Or, you can delete one by one resource by their name by this tutorial, run:

$ kubectl delete Memcachedopsrequest mcops-rotate-auth-generated mcops-rotate-auth-user -n demo
Memcachedopsrequest.ops.kubedb.com "mcops-rotate-auth-generated" "mcops-rotate-auth-user" deleted
$ kubectl delete secret -n demo mc-new-auth
secret "mc-new-auth" deleted
$ kubectl delete secret -n demo  memcd-quickstart-auth
secret "memcd-quickstart-auth" deleted

Next Steps