Set sylvaCA for ssl verification

What does this MR do and why?

Related reference(s)

Using a certificate signed by an external PKI is a supported feature in Sylva, but based on issue #1585 (closed), there are some problems accessing the UIs when keycloak is configured with an external certificate and other components are not (grafana in that case). Acces in Grafana UI via SSO is not possible and this error will occur failed to verify certificate: x509: certificate signed by unknown authority in grafana, because the secrets mounted for ssl contains only internal CA.

Grafana is not a particular case, I've noticed same issue in rancher and flux UIs.

2024-09-13T07:12:46.778Z        DEBUG   gitops  auth/init.go:77 OIDC config     {"IssuerURL": "https://keycloak.sylva/realms/sylva", "ClientID": "flux-webui", "ClientSecretLength": 32, "RedirectURL": "https://flux.sylva/oauth2/callback", "TokenDuration": "1h0m0s"}
Error: could not initialise authentication server: could not create auth server: could not create provider: Get "https://keycloak.sylva/realms/sylva/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority

2024/09/13 07:27:08 [DEBUG] 2024/09/13 07:27:08 http: superfluous response.WriteHeader call from github.com/rancher/rancher/pkg/version.(*versionHandler).ServeHTTP (version.go:53)
2024/09/13 07:27:08 [DEBUG] Create Token Invoked
2024/09/13 07:27:08 [ERROR] API error response 500 for POST /v3-public/keyCloakOIDCProviders/keycloakoidc?action=login. Cause: Get "https://keycloak.sylva/realms/sylva/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority

In order to fix it we have to mount sylva-ca.crt which contains the internal CA and also all the external CA that are used by the platform.

Additionally, I observed another issue in keycloak-add-truststore.sh because it will try to recreate keycloak-truststore configmap on each apply even if no changes were made and this stuck the deployment ( kustomization failed, configmap already exist and can not be recreated on apply.sh). To fix it, I've added a label with the checksum of ca.crt value on configmap and check it in order to know if a new certificate has been added or not and recreate it if is needed.

Close #1585 (closed).

Test coverage

Deploy the stack using external certificate for keycloak and mount sylva-ca.crt as additional-ca.

 openssl s_client -connect keycloak.sylva:443 --showcerts
CONNECTED(00000003)
---
Certificate chain
 0 s:C = RO, ST = Bucharest, L = Bucharest, O = Orange, OU = Sylva, CN = keycloak.sylva
   i:C = RO, ST = Romania, L = Bucharest, O = Orange, OU = Sylva, CN = Test-Sylva-CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep 12 12:44:37 2024 GMT; NotAfter: Sep 12 12:44:37 2025 GMT

openssl s_client -connect grafana.sylva:443 --showcerts
CONNECTED(00000003)
---
Certificate chain
 0 s:C = eu, O = Sylva, OU = DEV, CN = grafana.sylva
   i:C = eu, O = Sylva, OU = DEV, CN = Sylva CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Sep 20 08:14:11 2024 GMT; NotAfter: Dec 19 08:14:11 2024 GMT

 openssl s_client -connect rancher.sylva:443 --showcerts
CONNECTED(00000003)
---
Certificate chain
 0 s:C = eu, O = Sylva, OU = DEV, CN = rancher.sylva
   i:C = eu, O = Sylva, OU = DEV, CN = Sylva CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Sep 20 08:14:17 2024 GMT; NotAfter: Dec 19 08:14:17 2024 GMT

Access Grafana/Rancher UI via SSO login script:

Checking SSO auth Rancher
https://rancher.sylva/dashboard/auth/login?timed-out
Rancher
Redirect to SSO
Sign in to Sylva
https://keycloak.sylva/realms/sylva/protocol/openid-connect/auth?client_id=rancher&response_type=code&redirect_uri=https%3A%2F%2Francher.sylva%2Fverify-auth&scope=openid%20profile%20email&state=eyJub25jZSI6IlVJZlpRamVMdDhqMEQ0d0ciLCJ0byI6InZ1ZSIsInByb3ZpZGVyIjoia2V5Y2xvYWtvaWRjIn0
https://rancher.sylva/dashboard/auth/verify?state=eyJub25jZSI6IlVJZlpRamVMdDhqMEQ0d0ciLCJ0byI6InZ1ZSIsInByb3ZpZGVyIjoia2V5Y2xvYWtvaWRjIn0&session_state=d185f06b-572d-466d-8ff0-fd46ef89859a&iss=https%3A%2F%2Fkeycloak.sylva%2Frealms%2Fsylva&code=74273f38-2626-43d1-a5f0-4c7273154f13.d185f06b-572d-466d-8ff0-fd46ef89859a.b1b8391e-d133-45f2-9b1e-690d1398fd3a
True
Waiting to be redirect towards rancher UI home page
Redirect to rancher UI home page
https://rancher.sylva/dashboard/home
No workload cluster present on this configuration
Rancher SSO check done

Checking SSO auth Grafana
https://grafana.sylva/login
Grafana
Redirect to SSO
Sign in to Sylva
https://keycloak.sylva/realms/sylva/protocol/openid-connect/auth?access_type=online&client_id=grafana&redirect_uri=https%3A%2F%2Fgrafana.sylva%2Flogin%2Fgeneric_oauth&response_type=code&scope=openid+email+profile+offline_access+roles&state=5ZXqkNu0QLFLR6oYJ84s-c9Evd41yMFS1Gz4NVTWvgg%3D
https://grafana.sylva/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/?orgId=1
Waiting to be redirect towards grafana UI home page
Redirect to grafana UI home page
Home - Dashboards - Grafana
https://grafana.sylva/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/?orgId=1
Access dashboards
https://grafana.sylva/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/dashboards
Trying to access Grafana Overwiev dashboard
Grafana Overview - Dashboards - Grafana
https://grafana.sylva/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/d/6be0s85Mk/grafana-overview?query=&orgId=1
Grafana SSO check done

CI configuration

CI pipelines perform an update for both management and workload clusters, this update will NOT perform a ClusterAPI rolling upgrade (deletion and creation of new K8s nodes) by default.

For some cases, it may be relevant to perform more complex tests.

Theses features can be activated in an MR by adding one of these labels to the MR and will apply to the next pipelines.

• adding the label ci-featuretest-rolling-upgrade pipelines will perform a node rolling update in the -update jobs (without version upgrades)
• adding the label ci-featuretest-upgrade-from-1.1.1 pipelines will perform an upgrade from Sylva 1.1.1 to your dev branch (including a k8s version upgrade resulting in a node rolling update)