Issues applying and referencing CRDs from the same manifest project
Is there a recommended way to achieve this? Basically we just wanted to provision cert-manager
plus a default ClusterIssuer
resource:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: user@example.com
server: https://acme-v02.api.letsencrypt.org/directory
http01: {}
privateKeySecretRef:
name: letsencrypt-prod
We anticipated issues in applying all resources initially, but expected that everything would eventually reconcile properly after multiple attempts. What seems to happen instead is that if the agent can't apply all resources at once, it applies nothing. It also appears to never retry unless changes are made to the manifest repo.
It's not really clear to us what happens in the 3rd case (below) as our configuration is reported to be invalid, but a sync is also attempted. It must only be attempting to sync from the last manifest_projects
config block listed?
1st Attempt
In our first attempt we output all our resources from kustomize build .
to a directory and try to sync everything at once.
.gitlab/agents/staging/config.yaml
gitops:
manifest_projects:
- id: imp/sti/cluster-config
paths:
- glob: '/manifests/**/*.yaml'
$ kpt cfg tree manifests
manifests
├── [admissionregistration.k8s.io_v1_mutatingwebhookconfiguration_cert-manager-webhook.yaml] MutatingWebhookConfiguration cert-manager-webhook
├── [admissionregistration.k8s.io_v1_validatingwebhookconfiguration_cert-manager-webhook.yaml] ValidatingWebhookConfiguration cert-manager-webhook
├── [admissionregistration.k8s.io_v1_validatingwebhookconfiguration_ingress-nginx-admission.yaml] ValidatingWebhookConfiguration ingress-nginx-admission
├── [apiextensions.k8s.io_v1_customresourcedefinition_certificaterequests.cert-manager.io.yaml] CustomResourceDefinition certificaterequests.cert-manager.io
├── [apiextensions.k8s.io_v1_customresourcedefinition_certificates.cert-manager.io.yaml] CustomResourceDefinition certificates.cert-manager.io
├── [apiextensions.k8s.io_v1_customresourcedefinition_challenges.acme.cert-manager.io.yaml] CustomResourceDefinition challenges.acme.cert-manager.io
├── [apiextensions.k8s.io_v1_customresourcedefinition_clusterissuers.cert-manager.io.yaml] CustomResourceDefinition clusterissuers.cert-manager.io
├── [apiextensions.k8s.io_v1_customresourcedefinition_issuers.cert-manager.io.yaml] CustomResourceDefinition issuers.cert-manager.io
├── [apiextensions.k8s.io_v1_customresourcedefinition_orders.acme.cert-manager.io.yaml] CustomResourceDefinition orders.acme.cert-manager.io
├── [appdat-system_apps_v1_daemonset_ingress-nginx-controller.yaml] DaemonSet appdat-system/ingress-nginx-controller
├── [appdat-system_apps_v1_deployment_cert-manager-cainjector.yaml] Deployment appdat-system/cert-manager-cainjector
├── [appdat-system_apps_v1_deployment_cert-manager-webhook.yaml] Deployment appdat-system/cert-manager-webhook
├── [appdat-system_apps_v1_deployment_cert-manager.yaml] Deployment appdat-system/cert-manager
├── [appdat-system_batch_v1_job_ingress-nginx-admission-create.yaml] Job appdat-system/ingress-nginx-admission-create
├── [appdat-system_batch_v1_job_ingress-nginx-admission-patch.yaml] Job appdat-system/ingress-nginx-admission-patch
├── [appdat-system_rbac.authorization.k8s.io_v1_role_cert-manager-cainjector:leaderelection.yaml] Role appdat-system/cert-manager-cainjector:leaderelection
├── [appdat-system_rbac.authorization.k8s.io_v1_role_cert-manager-webhook:dynamic-serving.yaml] Role appdat-system/cert-manager-webhook:dynamic-serving
├── [appdat-system_rbac.authorization.k8s.io_v1_role_cert-manager:leaderelection.yaml] Role appdat-system/cert-manager:leaderelection
├── [appdat-system_rbac.authorization.k8s.io_v1_role_ingress-nginx-admission.yaml] Role appdat-system/ingress-nginx-admission
├── [appdat-system_rbac.authorization.k8s.io_v1_role_ingress-nginx.yaml] Role appdat-system/ingress-nginx
├── [appdat-system_rbac.authorization.k8s.io_v1_rolebinding_cert-manager-cainjector:leaderelection.yaml] RoleBinding appdat-system/cert-manager-cainjector:leaderelection
├── [appdat-system_rbac.authorization.k8s.io_v1_rolebinding_cert-manager-webhook:dynamic-serving.yaml] RoleBinding appdat-system/cert-manager-webhook:dynamic-serving
├── [appdat-system_rbac.authorization.k8s.io_v1_rolebinding_cert-manager:leaderelection.yaml] RoleBinding appdat-system/cert-manager:leaderelection
├── [appdat-system_rbac.authorization.k8s.io_v1_rolebinding_ingress-nginx-admission.yaml] RoleBinding appdat-system/ingress-nginx-admission
├── [appdat-system_rbac.authorization.k8s.io_v1_rolebinding_ingress-nginx.yaml] RoleBinding appdat-system/ingress-nginx
├── [appdat-system_v1_configmap_ingress-nginx-controller.yaml] ConfigMap appdat-system/ingress-nginx-controller
├── [appdat-system_v1_configmap_ingress-nginx-custom-add-headers.yaml] ConfigMap appdat-system/ingress-nginx-custom-add-headers
├── [appdat-system_v1_service_cert-manager-webhook.yaml] Service appdat-system/cert-manager-webhook
├── [appdat-system_v1_service_cert-manager.yaml] Service appdat-system/cert-manager
├── [appdat-system_v1_service_ingress-nginx-controller-admission.yaml] Service appdat-system/ingress-nginx-controller-admission
├── [appdat-system_v1_service_ingress-nginx-controller.yaml] Service appdat-system/ingress-nginx-controller
├── [appdat-system_v1_serviceaccount_cert-manager-cainjector.yaml] ServiceAccount appdat-system/cert-manager-cainjector
├── [appdat-system_v1_serviceaccount_cert-manager-webhook.yaml] ServiceAccount appdat-system/cert-manager-webhook
├── [appdat-system_v1_serviceaccount_cert-manager.yaml] ServiceAccount appdat-system/cert-manager
├── [appdat-system_v1_serviceaccount_ingress-nginx-admission.yaml] ServiceAccount appdat-system/ingress-nginx-admission
├── [appdat-system_v1_serviceaccount_ingress-nginx.yaml] ServiceAccount appdat-system/ingress-nginx
├── [default_certmanager.k8s.io_v1alpha1_clusterissuer_letsencrypt-prod.yaml] ClusterIssuer letsencrypt-prod
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-cainjector.yaml] ClusterRole cert-manager-cainjector
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-controller-certificates.yaml] ClusterRole cert-manager-controller-certificates
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-controller-challenges.yaml] ClusterRole cert-manager-controller-challenges
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-controller-clusterissuers.yaml] ClusterRole cert-manager-controller-clusterissuers
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-controller-ingress-shim.yaml] ClusterRole cert-manager-controller-ingress-shim
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-controller-issuers.yaml] ClusterRole cert-manager-controller-issuers
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-controller-orders.yaml] ClusterRole cert-manager-controller-orders
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-edit.yaml] ClusterRole cert-manager-edit
├── [rbac.authorization.k8s.io_v1_clusterrole_cert-manager-view.yaml] ClusterRole cert-manager-view
├── [rbac.authorization.k8s.io_v1_clusterrole_ingress-nginx-admission.yaml] ClusterRole ingress-nginx-admission
├── [rbac.authorization.k8s.io_v1_clusterrole_ingress-nginx.yaml] ClusterRole ingress-nginx
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_cert-manager-cainjector.yaml] ClusterRoleBinding cert-manager-cainjector
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_cert-manager-controller-certificates.yaml] ClusterRoleBinding cert-manager-controller-certificates
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_cert-manager-controller-challenges.yaml] ClusterRoleBinding cert-manager-controller-challenges
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_cert-manager-controller-clusterissuers.yaml] ClusterRoleBinding cert-manager-controller-clusterissuers
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_cert-manager-controller-ingress-shim.yaml] ClusterRoleBinding cert-manager-controller-ingress-shim
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_cert-manager-controller-issuers.yaml] ClusterRoleBinding cert-manager-controller-issuers
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_cert-manager-controller-orders.yaml] ClusterRoleBinding cert-manager-controller-orders
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_ingress-nginx-admission.yaml] ClusterRoleBinding ingress-nginx-admission
├── [rbac.authorization.k8s.io_v1_clusterrolebinding_ingress-nginx.yaml] ClusterRoleBinding ingress-nginx
└── [v1_namespace_appdat-system.yaml] Namespace appdat-system
agentk
pod logs
{"level":"info","time":"2021-02-12T21:32:09.119Z","msg":"Cluster successfully synced","mod_name":"gitops","project_id":"imp/sti/cluster-config"}
{"level":"info","time":"2021-02-12T21:32:10.560Z","msg":"Syncing","mod_name":"gitops","project_id":"imp/sti/cluster-config","skipHooks":false,"started":false}
{"level":"info","time":"2021-02-12T21:32:10.599Z","msg":"Adding resource result, status: 'SyncFailed', phase: '', message: 'the server could not find the requested resource'","mod_name":"gitops","project_id":"imp/sti/cluster-config","namespace":"default","kind":"ClusterIssuer","name":"letsencrypt-prod","phase":"Sync"}
{"level":"info","time":"2021-02-12T21:32:10.624Z","msg":"Updating operation state. phase: -> Failed, message: '' -> 'one or more synchronization tasks are not valid'","mod_name":"gitops","project_id":"imp/sti/cluster-config"}
{"level":"info","time":"2021-02-12T21:32:10.624Z","msg":"Synced","mod_name":"gitops","project_id":"imp/sti/cluster-config","resource_key":"certmanager.k8s.io/ClusterIssuer/default/letsencrypt-prod","sync_result":"the server could not find the requested resource"}
2nd Attempt
For our second attempt we output all our resources from kustomize build .
to a single file (since kustomize
intelligently orders the resources) hoping they would be applied in order and this would fix the problem.
Agent config remained the same, log output from agentk
was the same.
3rd Attempt
In our final attempt we output all our resources from kustomize build .
to a directory (same as 1st attempt) and updated the agent config to attempt to apply the CRDs separately from other resources.
.gitlab/agents/staging/config.yaml
gitops:
manifest_projects:
- id: imp/sti/cluster-config
resource_inclusions:
- api_groups:
- apiextensions.k8s.io/v1
kinds:
- '*'
paths:
- glob: '/manifests/**/*.yaml'
- id: imp/sti/cluster-config
resource_exclusions:
- api_groups:
- apiextensions.k8s.io/v1
kinds:
- '*'
paths:
- glob: '/manifests/**/*.yaml'
agentk
pod logs
{"level":"error","time":"2021-02-12T21:37:42.504Z","msg":"Failed to apply manifest projects configuration","mod_name":"gitops","error":"duplicate project id: imp/sti/cluster-config"}
{"level":"info","time":"2021-02-12T21:37:59.640Z","msg":"Syncing","mod_name":"gitops","project_id":"imp/sti/cluster-config","skipHooks":false,"started":false}
{"level":"info","time":"2021-02-12T21:37:59.684Z","msg":"Adding resource result, status: 'SyncFailed', phase: '', message: 'the server could not find the requested resource'","mod_name":"gitops","project_id":"imp/sti/cluster-config","namespace":"default","kind":"ClusterIssuer","name":"letsencrypt-prod","phase":"Sync"}
{"level":"info","time":"2021-02-12T21:37:59.702Z","msg":"Updating operation state. phase: -> Failed, message: '' -> 'one or more synchronization tasks are not valid'","mod_name":"gitops","project_id":"imp/sti/cluster-config"}
{"level":"info","time":"2021-02-12T21:37:59.702Z","msg":"Synced","mod_name":"gitops","project_id":"imp/sti/cluster-config","resource_key":"certmanager.k8s.io/ClusterIssuer/default/letsencrypt-prod","sync_result":"the server could not find the requested resource"}
In this case the logs seem to indicate that specifying the same manifest project with different resource inclusion/exclusion blocks is considered invalid. The thinking was that we could trick the agent into creating different reconciliation loops for the CRDs and resources so they could be synced independently.