Fix kas redis secret mounting in init-etc-kas

What does this MR do?

This MR fixes the kas redis secret mounting behaviour in init-etc-kas. From the kubectl dry-run in https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/12406463, we see that password_file is updated but the file in question is not mounted.

That is because the mountPath is pinned to sharedState. So 2 files get mounted whenever Redis is used: redis/sharedState-password and redis/redis-password.

The original change was introduced in !3544 (merged).

Test plan

1. Set up a cluster

kind create cluster --config examples/kind/kind-no-ssl.yaml

2. Update examples/kind/values-no-ssl.yaml

diff --git a/examples/kind/values-no-ssl.yaml b/examples/kind/values-no-ssl.yaml
index bf8373aef..619edd295 100644
--- a/examples/kind/values-no-ssl.yaml
+++ b/examples/kind/values-no-ssl.yaml
@@ -5,7 +5,20 @@ global:
   ingress:
     tls:
       enabled: false
-
+  redis:
+    host: localhost
+    sharedState:
+      password:
+        enabled: true
+        secret: gitlab-redis-cluster-shared-state-rails-credential-v1
+        key: password
+    kas:
+      password:
+        enabled: true
+        secret: gitlab-redis-kas-rails-credential-v1
+        key: password
+redis:
+  install: false
 # Specify NodePort for HTTP on NGINX

3. Start the cluster using:

helm upgrade --install gitlab .  --set global.hosts.domain=192.168.1.74.nip.io -f examples/kind/values-base.yaml  -f examples/kind/values-no-ssl.yaml

4. Run the following command to check the mounted volumes

`kubectl describe pod $(kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep kas | head -n 1)`

On master, the gitlab-redis-kas-rails-credential-v1 secret is not mounted. Instead the shared state secret is mounted:

  init-etc-kas:
    Type:                Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:       gitlab-kas
    ConfigMapOptional:   <nil>
    SecretName:          gitlab-gitlab-kas-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-kas-private-api
    SecretOptionalName:  <nil>
    SecretName:          gitlab-redis-cluster-shared-state-rails-credential-v1
    SecretOptionalName:  <nil>
    SecretName:          gitlab-redis-secret
    SecretOptionalName:  <nil>

On this branch:

We see that in the init-etc-kas volume, gitlab-redis-kas-rails-credential-v1 is mounted:

  init-etc-kas:
    Type:                Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:       gitlab-kas
    ConfigMapOptional:   <nil>
    SecretName:          gitlab-gitlab-kas-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-kas-private-api
    SecretOptionalName:  <nil>
    SecretName:          gitlab-redis-kas-rails-credential-v1
    SecretOptionalName:  <nil>
    SecretName:          gitlab-redis-secret
    SecretOptionalName:  <nil>

Note: we will get Warning FailedMount 18s (x8 over 82s) kubelet MountVolume.SetUp failed for volume "init-etc-kas" : secret "gitlab-redis-kas-rails-credential-v1" not found since there is no actual redis but that is not what we are testing for.

Related issues

Author checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• When ready for review, follow the instructions in the "Reviewer Roulette" section of the Danger Bot MR comment, as per the Distribution experimental MR workflow

For merge requests from forks, consider the following options for Danger to work properly:

• Consider using our community forks instead of forking your own project. These community forks have the GitLab API token preconfigured.
• Alternatively, see our documentation on configuring Danger for personal forks.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added/updated
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened -- NA since gitlab-kas redis is already configured separately (gitlab-com/gl-infra/scalability#2651 (comment 1699981991))
• Equivalent MR/issue for Gitlab Operator project opened (see Operator documentation on impact of Charts changes)
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.