Reevaluate manifest design

Summary

Over time, we have noticed that working with the manifests under config/ has proved challenging.

For example, #245 (closed) required significant research and implementation to get the CI process in-line with the user-documented installation steps. It involved teaching Kustomize how to make substitutions across many different objects because it didn't know how to by default.

Potential solutions

Helm

Other Operators appear to use Helm to generate the static manifest and provide a mechanism for tweaking the installation configuration. For example, the CertManager's kubectl documentation specifies:

By default, cert-manager will be installed into the cert-manager namespace. It is possible to run cert-manager in a different namespace, although you’ll need to make modifications to the deployment manifests.

Then, in their helm docs, they show that you can change configuration:

$ helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.5.3 \
  --set prometheus.enabled=false \  # Example: disabling prometheus using a Helm parameter
  --set webhook.timeoutSeconds=4s   # Example: changing the wehbook timeout using a Helm parameter

This solution would enable us to:

  • Remove our complicated kustomize/kustomizeconfig setup
  • Expand the amount of tweaks we support (like Deployment replicas, ServiceAccount names, etc.)
  • Provide another installation option that is widely used by the community

Acceptance criteria

  • A decision is made whether or not to refactor our manifest design
  • If the decision is made to refactor our manifest design, then a new issue is created outlining the decision and will be closed by an MR that implements that design