One other way of deleting CRDs was described here. This method still involves running custom kubectl commands against the cluster but the commands are run from a post-delete hook that spins up a kind: Job resource:
More info about about it. Some people were able to make use of post-delete hook to spin up jobs that create a resource kind: Job that will run the script for removing CRDs and Namespaces inside the container (reference). Finally the Job kills itself after completing execution by using a hook-delete-policy: :hook-succeeded in the Job definition.
The main difference of the aforementioned approach is that the main responsibility of deleting the CRDs stays with the chart maintainer, since the webhooks and Job resource would need to be setup there.
I also believe that a pre-delete with another Job will also be necessary to first remove the possibly existing Knative deployed services.
@Alexand I'm personally in favour of deleting by api group as @tkuah suggested. It seems to me that the only API groups we have to delete are exactly named as the chart we are uninstalling so it's a perfect match.
We could always deal with the issue you identify later if another API group is added that we think is more broad than the 2 charts we're uninstalling.
Ok. I guess I'm not all that worried about defensively programming against the case of someone installing a CRD into the istio or knative group and then uninstalling istio and knative and being surprised their CRD is gone. But I suppose it's possible.
I tend to agree it's a very edge case - probably solvable by noting in the uninstall pop-up that all CRDs belonging to api groups *.istio.dev and *.knative.dev will deleted.
My main concern is about version upgrades, since I believe it can easily happen that someone bumps a version and forgets to check on the uninstall cleaning and reinstalling. In my opinion, the most safe solution in all aspects is to pin the CRDs and find a way to read the CRDs inside the chart as a unit test. I guess it would cover all the possible problems.
Although, I understand that the API group list is not only smaller but less likely to change whilst the CRDs might be more likely to change. So maintaining the list of API groups might be easier. I'm happy to go with this solution and see how it goes.
When we implemented the Uninstall button for Knative, we started tracking a list of CRDs that we had to explicitly delete as post uninstall step, since the helm chart wouldn't remove them after Knative is uninstalled. See:
Although, this is a long list to be maintained. So back then, @tkuah suggested, as a follow-up issue, that we delete by CRD api-group instead of each CRD individually. This would reduce this maintained list drastically.
I'm happy to do this refactoring, however, I'd like your green light to go forward with this, since I don't know if you might have any different plans for this stuff.
@grzesiek , I don't think kubectl can delete everything under a group with a single command like kubectl delete group group_name --all, would be neat by the way. Although, one possible way is to fetch all the api resources in an api group then delete them:
kubectl api-resources --api-group='networking.istio.io'-o name | xargs kubectl delete crd
This way we only need to maintain the list of api groups. Is that also what you had in mind @tkuah ?
For that they're also suggesting that installation is done by using Helm only as a template generator, then following a kubectl apply -f on the generated file:
the helm template case is the main use case that must be preserved. It is far more important than Helm not deleting CRDs. If I have to trade one for another, I'll keep helm template with kubectl apply -f...
Here are the api-groups found from our Knative 0.7.0 chart:
No record is nil, which makes me believe that all CRDs belong to an api-group
# After downloading and extracting locally the Knative 0.7.0 chart:# wget https://storage.googleapis.com/triggermesh-charts/knative-0.7.0.tgz# open knative-0.7.0.tgz# cd ./knativerequire'yaml'# Istio api-groupsarray=[]YAML.load_stream(File.read('./templates/istio.yaml')){|doc|array<<doc}array.select{|i|i&.[]("kind")=="CustomResourceDefinition"}.map{|j|j["spec"]["group"]}.uniq!=>["networking.istio.io","rbac.istio.io","authentication.istio.io","config.istio.io"]# Knative api-groupsarray=[]YAML.load_stream(File.read('./templates/knative.yaml')){|doc|array<<doc}array.select{|i|i&.[]("kind")=="CustomResourceDefinition"}.map{|j|j["spec"]["group"]}.uniq!=>["networking.internal.knative.dev","serving.knative.dev","caching.internal.knative.dev","autoscaling.internal.knative.dev"]