WIP: Resolve "Mutual SSL Auth For Helm TIller"

What does this MR do?

As part of our move to support RBAC it will become increasingly important to lock down the permissions of Helm Tiller. As such we will use mutual SSL auth between Helm client and Tiller server with the keys only available to GitLab so users cannot authenticate to Tiller at all.

TODO

• Rebase and fix conflicts
• Tests passing
• Remove base64 logic (not needed if we quote env vars when using echo $FOO)
• Ensure we have test coverage (at least at behaviour level so extract will be covered)
• Extract class to create/sign certs
• Generate client key on the fly so we don't need to persist it
• Don't persist tiller key or cert either
• Handle the case where Tiller was deployed already and does not use Mutual SSL
• Persist certs in separate table (Helm has_one :ca_cert; has_one client_cert; has_one server_cert)
• Encrypt secret keys in the DB so no secrets are stored unencrypted on disk
• Fix spec/features/projects/clusters/applications_spec.rb somehow. Perhaps re-write it? Relying on polling and the way things are mocked in this test make it very flakey and difficult to troubleshoot.
• Squash commits
• Move all values.yml mounts to secrets and then add the certs/keys to this and then we can re-use the same method to pass this data through rather than env vars. Bonus we now have these values passed through for all applications including tiller which improves the inheritance similarities. We can also refactor so that merging in content to the values.yaml can be done in child objects using super to load the file then merging in rather than overwriting like we do in the runner now. We should use k8s secrets to pass the helm client key because we need to ensure that nobody gets hold of a helm client key during the install phase. If somebody can get the helm client key during the install phase then they can install whatever they like with tiller which then defeats the whole purpose of what this auth is trying to accomplish.
• Improve Certificate class. Reduce duplication in cert generation + improve name since it is a cert and key pair
• Make sure our CA certs and Tiller certs don't ever expire

Manual QA To Do

• Install tiller with code that does not set up mutual SSL, then upgrade code and install applications and see that this still works as expected (including JupyterHub)
• Install tiller + 1 application, then upgrade code and install the rest of the applications (including JupyterHub)

Are there points in the code the reviewer needs to double check?

• Security, encrypting data in the database, passing data to Helm

Why was this MR needed?

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Documentation created/updated
• [-] API support added
• Tests added for this feature/bug
• Conform by the code review guidelines
  • [-] Has been reviewed by a UX Designer
  • [-] Has been reviewed by a Frontend maintainer
  • Has been reviewed by a Backend maintainer
  • Has been reviewed by a Database specialist
• Conform by the merge request performance guides
• Conform by the style guides
• If you have multiple commits, please combine them into a few logically organized commits by squashing them
• Internationalization required/considered
• End-to-end tests pass (package-and-qa manual pipeline job)

What are the relevant issue numbers?

Closes #48098 (closed)