When creating a secret, RandomSecret cannot verify that secret exists

Summary

When randomsecret applies to an existing secret it updates the secret. This is not the expected behaviour and it can lead to differences between the password stored in vault and the password used by the unit (This is what happens to Keycloak after un upgrade for example).

Steps to reproduce

Apply randomsecret on an existing secret.

What is the current bug behavior?

A new version of the secret has been created and the value of the secret has changed.

What is the expected correct behavior?

The secret should not be modified since randomsecret does not specify a refresh period.

Details

Let's create a secret in Vault:

$ vault kv put secret/test foo=bar
== Secret Path ==
secret/data/test

======= Metadata =======
Key                Value
---                -----
created_time       2025-09-09T12:40:39.703617341Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Verify the value of the secret:

$ vault kv get secret/test
== Secret Path ==
secret/data/test

======= Metadata =======
Key                Value
---                -----
created_time       2025-09-09T12:53:29.880640503Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

=== Data ===
Key    Value
---    -----
foo    bar

Now let's call randomsecret to create a secret also named test:

$ cat > random-secret.yaml <<EOF
apiVersion: redhatcop.redhat.io/v1alpha1
kind: RandomSecret
metadata:
  name: test
  namespace: vault
spec:
  connection:
    tLSConfig:
      skipVerify: true
    address:  https://vault.vault.svc.cluster.local:8200
  authentication:
    path: kubernetes
    role: secret-writer
    serviceAccount:
      name: vault
  isKVSecretsEngineV2: true
  path: secret/data
  secretKey: foo
  secretFormat:
    passwordPolicyName: sylva-password-policy
EOF

Apply random-secret.yaml:

$ kubectl --kubeconfig management-cluster-kubeconfig apply -f random-secret.yaml
randomsecret.redhatcop.redhat.io/test created

We can see that a new version of the secret has been created and its value has changed when it shouldn't have:

$ vault kv get secret/test
== Secret Path ==
secret/data/test

======= Metadata =======
Key                Value
---                -----
created_time       2025-09-09T12:50:20.610288289Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

=== Data ===
Key    Value
---    -----
foo    a6dSUzBJDQ2ZUKGBinMCTST9pFS9D76S

Actually, the sevice account used by randomsecret to authenticate against vault must be bound to a vault policy allowing to check secrets so that vault-config-operator can verify that the secret exists, i.e. the policy must include the verbs list and read. This is not what the policy secret-writer does (cf. https://gitlab.com/sylva-projects/sylva-core/-/blob/main/kustomize-units/vault/vault.yaml?ref_type=heads#L170), i.e. only the operation write is allowed. So the permission denied shown in the logs of vault-config-operator:

$ kubectl --kubeconfig management-cluster-kubeconfig -n vault logs vault-config-operator-744464bcff-qgpt6
....
2025-09-09T12:50:20Z    ERROR   unable to check object existence at     {"controller": "randomsecret", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "RandomSecret", "RandomSecret": {"name":"test","namespace":"vault"}, "namespace": "vault", "name": "test", "reconcileID": "e0633606-266e-4c8d-ae62-c4a8f8da4212", "path": "secret/data/test", "error": "Error making API request.\n\nURL: GET https://vault.vault.svc.cluster.local:8200/v1/secret/data/test\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n\n"}
github.com/redhat-cop/vault-config-operator/api/v1alpha1/utils.(*VaultEndpoint).Exists
        /home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/utils/vaultobject.go:94
github.com/redhat-cop/vault-config-operator/controllers.(*RandomSecretReconciler).manageReconcileLogic
        /home/runner/work/vault-config-operator/vault-config-operator/controllers/randomsecret_controller.go:158
github.com/redhat-cop/vault-config-operator/controllers.(*RandomSecretReconciler).Reconcile
        /home/runner/work/vault-config-operator/vault-config-operator/controllers/randomsecret_controller.go:105
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227
2025-09-09T12:50:20Z    INFO    controllers.RandomSecret        unable to verify secret existence       {"instance": {"kind":"RandomSecret","apiVersion":"redhatcop.redhat.io/v1alpha1","metadata":{"name":"test","namespace":"vault","uid":"2be49f20-f1f5-430d-b3ca-13b67473df5e","resourceVersion":"181583256","generation":1,"creationTimestamp":"2025-09-09T12:50:20Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"redhatcop.redhat.io/v1alpha1\",\"kind\":\"RandomSecret\",\"metadata\":{\"annotations\":{},\"name\":\"test\",\"namespace\":\"vault\"},\"spec\":{\"authentication\":{\"path\":\"kubernetes\",\"role\":\"secret-writer\",\"serviceAccount\":{\"name\":\"vault\"}},\"connection\":{\"address\":\"https://vault.vault.svc.cluster.local:8200\",\"tLSConfig\":{\"skipVerify\":true}},\"isKVSecretsEngineV2\":true,\"path\":\"secret/data\",\"secretFormat\":{\"passwordPolicyName\":\"sylva-password-policy\"},\"secretKey\":\"foo\"}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"redhatcop.redhat.io/v1alpha1","time":"2025-09-09T12:50:20Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:authentication":{".":{},"f:path":{},"f:role":{},"f:serviceAccount":{}},"f:connection":{".":{},"f:address":{},"f:tLSConfig":{".":{},"f:skipVerify":{}}},"f:isKVSecretsEngineV2":{},"f:kvSecretRetainPolicy":{},"f:path":{},"f:secretFormat":{".":{},"f:passwordPolicyName":{}},"f:secretKey":{}}}}]},"spec":{"connection":{"tLSConfig":{"skipVerify":true},"address":"https://vault.vault.svc.cluster.local:8200"},"authentication":{"serviceAccount":{"name":"vault"},"path":"kubernetes","role":"secret-writer"},"path":"secret/data","secretFormat":{"passwordPolicyName":"sylva-password-policy"},"secretKey":"foo","isKVSecretsEngineV2":true,"kvSecretRetainPolicy":"Delete"},"status":{}}, "error": "Error making API request.\n\nURL: GET https://vault.vault.svc.cluster.local:8200/v1/secret/data/test\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n\n"}
2025-09-09T12:50:20Z    INFO    randomsecret-resource   validate update {"name": "test"}

Fix

Add the capability to read/list secrets to the vault policy secret-writer (BTW, the policy should thus be renamed as secret-rw):

path "secret/data/*" {
  capabilities = [ "create", "update", "read", "delete", "list" ]
}

path "secret/metadata/*" {
  capabilities = [ "delete" ]
}

# required to create and use the password policy

path "sys/policies/password/sylva-password-policy*" {
  capabilities = ["create", "read", "sudo", "update"]
}

path "/sys/policies/password/+/generate" {
  capabilities = [ "read" ]
}

After updating secret-writer as above, randomsecret can verify that the secret exist and, if so, the secret is not modified. This is shown in the logs of vault-config-operator after applying randomsecret:

$ kubectl --kubeconfig management-cluster-kubeconfig -n vault logs vault-config-operator-665cdd48b8-jxk7z
....
2025-05-27T14:17:48Z    INFO    authenginemount-resource        default {"name": "test"}
2025-05-27T14:17:48Z    INFO    randomsecret-resource   validate create {"name": "test"}
2025-05-27T14:17:48Z    DEBUG   Creating new client     {"controller": "randomsecret", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "RandomSecret", "RandomSecret": {"name":"test","namespace":"vault"}, "namespace": "vault", "name": "test", "reconcileID": "83c14fda-c040-47d3-b756-30f934008b7b"}
2025-05-27T14:17:48Z    INFO    controllers.RandomSecret        no refresh period is defined and Vault secret already exists - nothing to do    {"name": "test"}
2025-05-27T14:17:48Z    INFO    randomsecret-resource   validate update {"name": "test"}

cc @tmmorin

Edited Sep 10, 2025 by Pierrick Seite