[Spike] Consider validating the Helm template in AdmissionWebhook

Summary

Context: thread

!221 (merged) added support for creating Events, including an event to communicate that the Helm template failed to render.

It was decided in that MR to not requeue a reconcile loop because it would be considered wasteful since the Helm template will not render properly until the configuration is fixed. And fixing the configuration would trigger a new loop anyway.

However, this conceptually breaks the controller pattern where any error should be returned to the reconciler which will then requeue a loop in the future.

Proposal

Ideally, we could use our AdmissionWebhook to validate the Helm template before allowing it to be reconciled. This way, the error could be returned during kubectl apply -f mygitlab.yaml.

At the moment, the webhook just does some basic checks like ensuring that spec.chart.version is set. However, this would require importing the gitlab package so that the template could be rendered, which would lead to an import loop.

References

• https://v1-0-x.sdk.operatorframework.io/docs/building-operators/golang/webhooks
• https://book.kubebuilder.io/cronjob-tutorial/webhook-implementation.html
• https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers

Acceptance criteria

• Identify if rendering the template in the AdmissionWebhook is appropriate
  • While technically feasible from a code perspective, this won't be appropriate at this time because rendering the Helm template can often take more than 30 seconds, which exceeds the maximum allowed limit on AdmissionWebhook timeouts
• If the above is not appropriate, attempt to identify an alternative approach
  • The only alternative we're aware of at this time is to move away from the dependency on the Helm chart, which is a much longer-term path that we'll need to evaluate at another time (gitlab-org/distribution/team-tasks#990 (closed)).

See !344 (closed) for the attempted implementation. While the full implementation didn't work, I did extract some improvements into !345 (merged).