[WIP] THREESCALE-11020 Redis TLS certs and keys for porta and backend

[valerymo]

Jira: https://issues.redhat.com/browse/THREESCALE-11020

Add a way for the user to provide Redis TLS certs and keys for porta and backend

This PR enables Porta and Apisonator to load TLS configuration details for connecting to Redis. It introduces environment variables to specify the locations of the certificate files and indicate whether TLS mode is enabled.

For additional information, please refer to the Jira ticket and the documentation files included as part of this PR.

Validation

1. Install Redis Server for Test

- Create testing project: ``` export NAMESPACE=3scale-test oc new-project $NAMESPACE ``` - Copy the entire scripts block provided below, open your terminal and paste the script into the command line. This will create the Redis server, including following resources: - Secret: redis-tls-secret - ConfigMap: redis-config-redis - Deployment/pod: redis - Service: redis

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: redis-tls-secret
  namespace: 3scale-test
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lVYS9FaGY1UDR3WWVnYVZhaGpqT0R6ME1QVTJnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUl0eld6OHcrTDBOYXAzN0FIcXR4Rjh6OEdXRFROUnoKOUxVa0JYcXU1SkZhc1lxejYrZ2k3S3dsWUlTeEFwVzd4NXprWDZ4bjBCeGxsMXVNdzExZHVyeDJNUjhlYXFxQQpxbElFVzZwTU1kTGNndkhJUit0TGhLRFVvaVpsajdKRkk1MGYvYjhUaXRKVTByY1dhYmd4SVA4QnlINUdxSkR2CmtFZGgvNC9SbTlYeWU4SmhuMEVqRzdPUWxTM1MvOGJHSGpoV2Zjdys2QnJidDI5TE1aTzdJRmFlRVh4azRaQjAKS1VNODJ0TFhxV1VwaUdxQUhuWncrZUtBSlZDOFBzUzB5NU9aTGt4TE9GUlJnajJRYkF3TlZ1emxBN0VMQ0U4dApyVkx3OWV3aEhsVkhUcVZ0UXYwNWZVb2ZqOURkb3Yxd3F2OG1uZXJ3OHBnRVR3Yi9RemY1VGRjQ0F3RUFBYU5UCk1GRXdIUVlEVlIwT0JCWUVGRGpocmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFDdCtPNTZnanZVTUo5MFdaekFFcEFLcTNFTDY3c1I4NXlYam5EeXZML1lRRjZZSzRhUmVaZ1JICms5WCs2cDNyd2tsQmF6MzFFTFNFSGlPUkVTWDJzNW9WSVlId01SNUpsOE5VZTB6NGVBZlNNcTNtQmFuYWxneWYKQktROHZRc1UxTGVJQkpGRVJ5dG5aVE03VThFbFVsUkZSZzAzVUcyVG5ibGNxalFRa1lTMmZYT3M1czBlMHI2eQpaektXVWZhemtzTlkzSHlUcUxRTDlEZUM5STd6R0s2c2E5dHhueTFOM0I5bWJHZ1l3dlFZNDlvTm1kem9oWW16CkxmVVZJeERSWUcwNnI1UldoTTVtZDhZNmJUSGo1TExHT1A2YlEvYWJteXZxbFpuVnZjWTZaandKUzhNWVJJSk8KMmxrczl3cXRjeENla3VJZFBrOUNsNlVBZ3E5TkJoVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBakNDQWVxZ0F3SUJBZ0lVYUIyZGNLRXJuRHM0MEVqMitwT210bmpRQytVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQVAxN2tGaHBRdDhpcUlRWEJOUEozbHBrOFFGUWJnUncKc2VlQkNoa1RTYTd5S0hvQS9VTHVHY3QzZHVubWZPeWdNeEhueW9LUmR0NDVnWWlDOGNnQjE4OWRDRHo2cy84ZAp2TUdoUFUrblhkOWxyVHNtYXpLNW9McnVETmI3TXhBVDFVdjNIMXkvbFErY2tPTkttLzVhWndoNnFxL25XVFZtCkdOZUVGcDJyNUczTkVXcTJXWm1KM0RCc3pMeUdFciszdUlJbFVsaUk2bXRZUGRRd1QwR3RqTDJpVldRVUd2Q2sKYWpHclNJQUVETVFCdFNYeE1yOTlzS2VhS21iVDJ1L09BOEJjYVlqbXdZVnN4VDIzczR2UFltaUhjTFVjVFE4MApTblBLSW03NDJrTzhLZE5Id1hjN1BpejhUTVB0WTU1QkU5ZlBIaGdueVNBMVZUeWNQR1cyZEMwQ0F3RUFBYU5DCk1FQXdIUVlEVlIwT0JCWUVGT3BwbncveVd2dFo2L0llVzhUUlhXeG1uMUtXTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCWnhOZGpZR1Y4M1RBUwp6amMyNjZnc3gwRTB0cTRVUnRGWHdFeXZTYkxvY3BmK1V2OCtmTFBYZGF0WG1lRm5MdlJZbXg0VGR1UlltMFBYCk1BRUpqcEE3UXdlM1NsSmd5R2loNk1PVVdIRFdNZnhQZWJQWUh5ZFQ3TXBKVVJNWkxjZ0tvWlVENWNKdDc1bVIKMk51Z3BvdDdxQnYybzZGdmI5cmpNYSt1WFdJUEdSekhidjJ6eVFSR1J3SUpZcitKaDNWbXcyRSt0Vit0VEUwQQovWlBxOTIxYnlWYkREZHI2aDMyUnRRN1NJUnAwLzBnWFZablY3NTB2Rm9Ub2xOK2wwLzJOeEVCWmZNSzVFcEdkCkViSFFvN2dETHg0REljZlhqZVlxUkFuYjNLbzhXK0ZrQS8wQUdQS2NSN2RIL2daVnFHMngrRk5QazQ4YzBDQkwKR0R2RjlRYncKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRDllNUJZYVVMZklxaUUKRndUVHlkNWFaUEVCVUc0RWNMSG5nUW9aRTBtdThpaDZBUDFDN2huTGQzYnA1bnpzb0RNUjU4cUNrWGJlT1lHSQpndkhJQWRmUFhRZzgrclAvSGJ6Qm9UMVBwMTNmWmEwN0ptc3l1YUM2N2d6Vyt6TVFFOVZMOXg5Y3Y1VVBuSkRqClNwditXbWNJZXFxdjUxazFaaGpYaEJhZHErUnR6UkZxdGxtWmlkd3diTXk4aGhLL3Q3aUNKVkpZaU9wcldEM1UKTUU5QnJZeTlvbFZrRkJyd3BHb3hxMGlBQkF6RUFiVWw4VEsvZmJDbm1pcG0wOXJ2emdQQVhHbUk1c0dGYk1VOQp0N09MejJKb2gzQzFIRTBQTkVwenlpSnUrTnBEdkNuVFI4RjNPejRzL0V6RDdXT2VRUlBYeng0WUo4a2dOVlU4Cm5EeGx0blF0QWdNQkFBRUNnZ0VBRG5vaktWbUJyenJNZ3hiSmVNc2J2dS9xNzlkSElVdktiVjFxVlRwTHlBa2UKbExFL3hiWFJsVlJTWDFPQnFRWVJSS0dIYUdPa2RWYTFkalY4VjU3N1UyV04xZVcvcC85cnkyZEpHQ2FIN3YxZwpvbk0wUmlaaDdxc3Y0b3RnUkRmTnc5UHVYNTYxaGJtOGNLN1BML3k3eTdrdHpIUWJIVGlpakpTSHNpT2lIVDh1CitWc3oySzZmSnhHNUE2SE03bldvdlBuK1VydFd4OGd4OEJNQ1FOa3dUVnQvVmxzcE1wcUo1czdqeUNncGFQN2MKV0wrWnh2K2xUcFUwcnpJUTFZMWYvL0RSeUI3OVEycUl0dVhEdkJnNnI5ckExbytSV0JuNnA3WVA0T21SUE4vZQpiMVZZcG14c1BBL2ppQkhsRVFPZVJWbEVSYzhJSGVib2IxZFZ1Rm15b1FLQmdRRC9LeFlvVzNxaGNmUEYya2EvCmliZTkvKzlyQzNlTkdHUGwyaDVCeDJwK3k4dnlQb3NabmZ5STRKRzNtcXpVS1luSHFPVzhpTVpnR2ZOUlRNdVQKeVFSQitsVlFldHpOcDJUZCt4dWlCMUJmWHIyanowcVZvV3F4dFhrMklRWXFJN2NMSGZVZ2dGeUtsL2lRRGZlOQpwd0YwZ3c5Nm1vT012VWp1S1FmZjh2RTYxUUtCZ1FEK1R4SWZtc1VyY2VFdmpkeUdhNllweTBZcEx2OVhnSk0rCkxIc1ZmVTgzNXNWeVJaUDZrV1EwSGwyVCtHMDBqcStsU20vQ296cWNZeE5ObmtGMEhOdEw2SlJJWTBZZEExSFUKSmc1eTY3U2w0U0hHeWdYNTdrZEE4ajhyS1I4UXpmQ0RHT3Q5d2NwNjZ4MVlMSWEwRDYvV2h3elFOeXl3QzdnTwoxS3dvZlhyUCtRS0JnRzU4WUk2KzlYMWNVdnBUaGhpL2IvRDBGZDNhekR3cTJHNlpJRXJKSndLYUNjZnRidHQ3CnZmSWlrdFhXUW9sbkp3SnR6blB4SVR4UllEck9yc05oNGRjVHBzYy9POFpNZWU5b0lGSHJLdER3dTlwbkVsdHgKMWpuMll2S2VJQVkxQ3Jma2s5UXI0R1llWVlFMm14UGljVTNheGVRSGJYaU9LVHIrUnl1Z0RQVzFBb0dBZm5vdwoxMHNRR0tWUWkyZ1FiMElHcCs2Uy9GU0ZaYTFxalpkdHQ2aFV4OGFjR0ZNR1g2NERtZkFvTmpsdGhxQVlOeXFvCkhyTXpxU2VWS0JzM0RscHpybk1EbkdUVE1BYkFvYlF6cDNBV3JoRWp6VXdZWU03aTNTZ2R4b2R6RGRaK2NaVHAKT2VneG5hUmxPYjhiVjE0ZDQ2SFMrNU1WUkpEdmYyRENKbmtScFhFQ2dZRUFnVFB3MlVaSkJRVVdGc0NVRjczdgpGcWZSbzdZYmtweDAxcFlBcmV0ak41TTR6ZHZRUlZPQlFmVGRJTU9SbFJ4OVBnRitYL3UrT0szUzB1aS8xdWlECjhYU2QvMXFjS3ZOdUJ6ZnZNdTVGeVdKRHNCUGJaUHZuWnEzdTM5WktaYTNXNG92UE9VWlFERGlCTHNhM21CekwKcFdYNWFyVHNTazRSSDExQ28zRTN0akU9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: Opaque
EOF


cat << EOF | oc create -f -
apiVersion: v1
data:
  redis.conf: |+
    # redis.conf
    bind 0.0.0.0
    protected-mode no
    port 6379
    tls-port 6380
    tls-cert-file /etc/redis/certs/redis-server.crt
    tls-key-file /etc/redis/certs/redis-server.key
    tls-ca-cert-file /etc/redis/certs/ca.crt
    tls-auth-clients yes
    stop-writes-on-bgsave-error no
    save ""
kind: ConfigMap
metadata:
  name: redis-config-redis
EOF


cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: quay.io/fedora/redis-6
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/redis.conf
          subPath: redis.conf
        - name: redis-tls-volume
          mountPath: /etc/redis/certs
          readOnly: true
        command: ["/bin/sh", "-c", "redis-server /etc/redis/redis.conf"]
      volumes:
      - name: redis-config-volume
        configMap:
          name: redis-config-redis
      - name: redis-tls-volume
        secret:
          secretName: redis-tls-secret
EOF


cat << EOF | oc create -f -
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - port: 6379         # Non-TLS (unencrypted) port
      targetPort: 6379
      name: redis
    - port: 6380         # TLS port
      targetPort: 6380
      name: redis-tls
  selector:
    app: redis
  type: NodePort 
EOF

• Expecting results example: redis server pod is running and service available

$ oc get svc
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
redis   NodePort   172.30.56.166   <none>        6379:31290/TCP,6380:32389/TCP   10m

$ oc get pod
NAME                     READY   STATUS    RESTARTS   AGE
redis-5dc466fc8b-764hl   1/1     Running   0          10m

2. Certificates preparing

- Create CA, Client and Server Certificates, using Server IP as Common Name (CN): - Create directory `Certs`, `cd Certs`, and run following commands to create server and client certificates, that will be used for test.

openssl genpkey -algorithm RSA -out ca.key
openssl req -x509 -key ca.key -out ca.crt -days 365 -subj "/CN=172.30.56.166"

openssl genpkey -algorithm RSA -out redis-client.key
openssl req -new -key redis-client.key -out redis-client.csr -subj "/CN=redis-client.example.com"
openssl x509 -req -in redis-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-client.crt -days 365

openssl genpkey -algorithm RSA -out redis-server.key
openssl req -new -key redis-server.key -out redis-server.csr -subj "/CN=172.30.56.166"
openssl x509 -req -in redis-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-server.crt -days 365

• Expected files to be created:

Certs $ ls
ca.crt			ca.srl			redis-client.csr	redis-server.crt	redis-server.key
ca.key			redis-client.crt	redis-client.key	redis-server.csr

3. Update Redis Server with new server certificate

• Update redis-tls-secret secret, using new created:

  • ca.crt
  • redis-client.crt
  • redis-client.key

• Restart redis pod

4. Install 3scale

#### 4.1. Create Redis secrets for 3scale

- Below is a script to create the system-redis and backend-redis secrets with dummy client certificates. In the next step, we will replace these dummy certificates with valid client certificates, using the UI for convenience. The valid certificates will be sourced from the files created in the previous section. Additionally, we will update the Redis server URL to reflect the service IP of our Redis server.

You may prepare the secrets in whichever way is most convenient for you.

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: system-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: system
    app: 3scale-api-management
    threescale_component: system
data:
  SENTINEL_HOSTS: ''
  SENTINEL_ROLE: ''
  SSL_CA: cmVwbGFjZW1lCg==
  SSL_CERT: cmVwbGFjZW1lCg==
  SSL_KEY: cmVwbGFjZW1lCg==
  URL: cmVkaXM6Ly8xNzIuMzAuMjA3LjE5MTo2MzgwLzE=
type: Opaque
EOF

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: backend-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: backend
    app: 3scale-api-management
    threescale_component: backend
data:
  REDIS_STORAGE_URL: cmVkaXM6Ly8xNzIuMzAuMjA3LjE5MTo2MzgwLzA=
  REDIS_QUEUES_SENTINEL_HOSTS: ''
  REDIS_STORAGE_SENTINEL_ROLE: ''
  SSL_CA: cmVwbGFjZW1lCg==
  SSL_CERT: cmVwbGFjZW1lCg==
  SSL_KEY: cmVwbGFjZW1lCg==
  SSL_QUEUES_CA: cmVwbGFjZW1lCg==
  SSL_QUEUES_CERT: cmVwbGFjZW1lCg==
  SSL_QUEUES_KEY: cmVwbGFjZW1lCg==
  REDIS_QUEUES_URL: cmVkaXM6Ly8xNzIuMzAuMjA3LjE5MTo2MzgwLzE=
  REDIS_QUEUES_SENTINEL_ROLE: ''
  REDIS_STORAGE_SENTINEL_HOSTS: ''
type: Opaque
EOF

• Update Client Certificates in system-redis and backedn-redis secrets via UI. The following tables are for matching data field names with the certificate files created before:

• Secret: system-redis

Data field	Certificate file name
SSL_CA	ca.crt
SSL_CERT	redis-client.crt
SSL_KEY	redis-client.key

• Secret: backend-redis

Data field	Certificate file name
SSL_CA	ca.crt
SSL_CERT	redis-client.crt
SSL_KEY	redis-client.key
SSL_QUEUES_CA	ca.crt
SSL_QUEUES_CERT	redis-client.crt
SSL_QUEUES_KEY	redis-client.key

Please note:

• We are using a common CA for both the Redis server and client certificates.
• We are using the same client certificates for both the Redis system and the backend, including QUEUES.
• Please don't forget to update the Redis IP in the URLs to match the service IP and CN.
• Use secure Redis URLs. They should look like rediss://:6380/0, where rediss indicates a secure connection and port 6380 is used for secure Redis connections. For example:
  • REDIS_QUEUES_URL: rediss://172.30.56.166:6380/1
  • REDIS_STORAGE_URL: rediss://172.30.56.166:6380/0

4.2. Create s3-credentials secret

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: QUtJQVY2SVpYMk9ZQ09OWERFSlkK
  AWS_SECRET_ACCESS_KEY: aU5VbWdZY3hjSDF3azBlUlB0SytmTERHVVMvU0hxM1pKNVBlQy9xYQo=
  AWS_BUCKET: dm1vY2NzZjZxOGxyZWRoYXRyaG9hbW9wZXJhdG9ydGhyZWUtYW9mZwo=
  AWS_REGION: ZXUtd2VzdC0xCg==
type: Opaque
EOF

4.3. Run Install and Downloads commands

• To allow using External Redis:

export PREFLIGHT_CHECKS_BYPASS=true  

• Run Install and Downloads commands
  Please make sure you are in 3scale-operator directory, and run following commands:

make install
make download

4.4. Create APIManager CR and Run Operator

cat << EOF | oc create -f -
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  redisTLSEnabled: true
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: apps.vmo01.xcze.s1.devshift.org
  externalComponents:
    backend:
      redis: true
    system:
      redis: true 
EOF

• run Operator to install 3scale:

  make run
  

5 Check results and Troubleshooting

5.1 Expecting results

- All deployments and pods are UP, including those modified in the PR: - backend-cron - backend-listener - backend-worker - system-app - system-sidekiq

  oc get pod
NAME                                     READY   STATUS      RESTARTS        AGE
apicast-production-657f7f48bb-tm5tr      1/1     Running     0               8m44s
apicast-staging-d48c665f4-fmz5v          1/1     Running     0               8m44s
backend-cron-7fd9d6f445-qnsfv            1/1     Running     0               9m24s
backend-listener-58c5df7446-rvb4m        1/1     Running     0               9m22s
backend-worker-6989989f84-mb5j2          1/1     Running     0               9m22s
redis-5dc466fc8b-vxpbh                   1/1     Running     0               67m
system-app-5f6786dc98-fl59p              3/3     Running     0               8m11s
system-app-post-lglpl                    0/1     Completed   0               6m29s
system-app-pre-59fck                     0/1     Completed   0               6m54s
system-memcache-7f56b677dd-ttdzr         1/1     Running     0               9m23s
system-mysql-6547d59bb9-nt8rp            1/1     Running     0               9m24s
system-searchd-669dc7599c-d76pq          1/1     Running     0               9m22s
system-searchd-manticore-reindex-kq2pz   0/1     Error       0               8m45s
system-searchd-manticore-reindex-pvggx   0/1     Completed   0               8m22s
system-sidekiq-7796796778-t767h          1/1     Running     0               8m42s
throwaway-redis                          1/1     Running     0               110m
zync-77555479f-w6ff9                     1/1     Running     0               8m44s
zync-database-df8764bd4-krkp7            1/1     Running     0               8m44s
zync-que-844ff6987b-q26b2                1/1     Running     2 (8m34s ago)   8m44s

 oc get deploy
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
apicast-production   1/1     1            1           12m
apicast-staging      1/1     1            1           12m
backend-cron         1/1     1            1           13m
backend-listener     1/1     1            1           13m
backend-worker       1/1     1            1           13m
redis                1/1     1            1           5h43m
system-app           1/1     1            1           12m
system-memcache      1/1     1            1           13m
system-mysql         1/1     1            1           13m
system-searchd       1/1     1            1           13m
system-sidekiq       1/1     1            1           12m
zync                 1/1     1            1           12m
zync-database        1/1     1            1           12m
zync-que             1/1     1            1           12m

If you see Redis related errors, please see next section for Troubleshooting

5.2. Troubleshooting

If you encounter issues with the Redis server, one approach is to test the connection using the throwaway-redis pod, which includes redis-cli . This simplifies the process of checking TLS connections to Redis.

5.2.1 Using the throwaway-redis Pod

In this example, we assume that the throwaway-redis pod is not yet present in the 3scale-test project. Since we used the export PREFLIGHT_CHECKS_BYPASS=true environment variable earlier, the pod needs to be created; follow the steps below:

1. Create the throwaway-redis pod:
   • Stop the Operator: Press Ctrl+C.
   • Unset the preflight environment variable:

     unset PREFLIGHT_CHECKS_BYPASS
     

   • Restart the Operator:

   make run
   

   • In a separate terminal, check that the throwaway-redis pod is created.
   • Stop the Operator again once the pod is created.
   • Set the preflight bypass variable:

   export PREFLIGHT_CHECKS_BYPASS=true
   

   • Restart the Operator:

   make run
   

2. Copy certificates to the throwaway-redis pod:
   • Navigate to the directory where you saved the certificate files, then run the following commands to copy them to the pod:

     oc cp ca.crt throwaway-redis:/tmp
     oc cp redis-client.crt throwaway-redis:/tmp
     oc cp redis-client.key throwaway-redis:/tmp        
     

3. Login to the throwaway-redis pod and test the Redis connection using redis-cli:
   • Use the following command to access the throwaway-redis pod:

     oc rsh throwaway-redis
     

   • Once inside the pod, verify that the certificates are correctly copied:

     sh-5.1$ ls -l /tmp
     total 12
     -rw-r--r--. 1 redis root 1123 Dec 2 14:40 ca.crt
     -rw-r--r--. 1 redis root 1115 Dec 2 14:40 redis-client.crt
     -rw-------. 1 redis root 1704 Dec 2 14:40 redis-client.key     
     

- Now, attempt to connect to the Redis server using the redis-cli command with TLS enabled:

  ```
  sh-5.1$ redis-cli -h 172.30.56.166 --tls -p 6380 --cert redis-client.crt --key redis-client.key --cacert ca.crt ping   
  ```
  If successful, you should see the response:
  ```
  PONG
  ``` 
- Possible Error:
If you encounter the following error:
  ```
  Could not negotiate a TLS connection: Invalid CA Certificate File/Directory    
  ```
    - Resolution:  
Ensure that the certificates on both the server side and in the throwaway-redis pod are up-to-date and correctly populated.

4. Check non-TLS communication from the throwaway-redis pod:
   You can also check the Redis connection without TLS to verify basic connectivity. Run the following:

   oc rsh throwaway-redis
   cd /tmp
   sh-5.1$ redis-cli -h 172.30.56.166 -p 6379 ping 
   

If the connection is successful, you should see:
PONG
If the non-TLS connection works, but the TLS connection does not, consider updating the server certificates. To do this:
- Update the Redis testing server's secret: redis-tls-secret.
- Recreate the Redis testing server pod:
redis-5dc466fc8b-vxpbh 1/1 Running 0 4s

1. Recheck the TLS connection from throwaway-redis pod:
   After updating the server certificates and recreating the Redis pod, attempt the TLS connection again from the throwaway-redis pod. This time it looks good:
   sh-5.1$ redis-cli -h 172.30.56.166 --tls -p 6380 --cert redis-client.crt --key redis-client.key --cacert ca.crt ping
   The expected response should now be:

PONG

5.2.2 More things to check and useful commands

• check Logs in Init container of system-sidekiq pod:

  oc logs system-sidekiq-7796796778-sw7p4 -c check-svc
  Connected to rediss://172.30.56.166:6380/1
  Connected to rediss://172.30.56.166:6380/1
  Connected to rediss://172.30.56.166:6380/1
  Connected to rediss://172.30.56.166:6380/1
  

• Connect to Init container of system-sidekiq pod:

  kubectl exec -it  system-sidekiq-7796796778-sw7p4 -c check-svc  -- /bin/sh
  sh-4.4$ 
  
  sh-4.4$ env |grep tls   
  BACKEND_REDIS_PRIVATE_KEY=/tls/backend-redis-private.key
  REDIS_CLIENT_CERT=/tls/system-redis/system-redis-client.crt
  REDIS_CA_FILE=/tls/system-redis/system-redis-ca.crt
  BACKEND_REDIS_CA_FILE=/tls/backend-redis-ca.crt
  REDIS_PRIVATE_KEY=/tls/system-redis/system-redis-private.key
  BACKEND_REDIS_CLIENT_CERT=/tls/backend-redis-client.crt
  
  sh-4.4$ env |grep REDIS_SSL
  REDIS_SSL=1
  BACKEND_REDIS_SSL=1
  

6 Documentation, Notes

1. Please see documentation updated - [operator-user-guide - Setting Redis TLS Environment variables](https://github.com/valerymo/3scale-operator/blob/THREESCALE-11020-2/doc/operator-user-guide.md?plain=1#L957)

2. We provided notes for testing for APIManager CR options when redisTLSEnabled: true and both redis externalComponents are true.

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  redisTLSEnabled: true
  externalComponents:
    backend:
      redis: true
    system:
      redis: true     
  ```
</details>

[openshift-ci]

@valerymo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-e2e	1347947	link	true	/test test-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.