Use the most restrictive SCC possible per component

Summary

Adds separate RBAC objects for nonroot and anyuid ServiceAccounts so we can mix and match them across the GitLab components, using the most restrictive policy available for each component.

In summary:

• Components now use nonroot by default, rather than anyuid
• Webservice still gets to use anyuid because it needs the ability to issue KILL signals
• In the future, we can likely default components to restricted when gitlab-org/charts/gitlab!2369 (closed) is merged which will allow us to unset securityContext and allow OpenShift to assign random UIDs

Related to #120 (closed)

Testing

• CI pipeline: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/pipelines/599749389
• Unit tests: 66629240
• Manual validation: !447 (comment 912073411)

$ TAG=rework-scc-permissions task build_operator_openshift

$ kubectl apply -f .build/operator-openshift.yaml

$ cat mygitlab.yaml
apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
 name: gitlab
spec:
 chart:
   version: "6.1.2"
   values:
     global:
       hosts:
         domain: your.domain
         externalIP: your.static.external.IP
     certmanager-issuer:
       email: your@email.com

$ kubectl apply -f mygitlab.yaml -n gitlab-system

Confirming ServiceAccount per component:

$ k get pods -n gitlab-system -ojson | jq '.items[] | {"name": .metadata.name, "sa": .spec.serviceAccountName}'

{
  "name": "gitlab-controller-manager-69fc4655c7-pblp9",
  "sa": "gitlab-manager"
}
{
  "name": "gitlab-gitaly-0",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-gitlab-exporter-545d88d5f6-wvdz4",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-gitlab-shell-587fcf48d-cj8xd",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-gitlab-shell-587fcf48d-trfkr",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-minio-0",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-nginx-ingress-controller-795897994-8wnpz",
  "sa": "gitlab-nginx-ingress"
}
{
  "name": "gitlab-nginx-ingress-controller-795897994-njj4l",
  "sa": "gitlab-nginx-ingress"
}
{
  "name": "gitlab-nginx-ingress-defaultbackend-665498cd7d-vhndb",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-operator-cert-manager-774896fdf4-jkzjd",
  "sa": "gitlab-operator-cert-manager"
}
{
  "name": "gitlab-operator-cert-manager-cainjector-558dff4885-hdvj5",
  "sa": "gitlab-operator-cert-manager-cainjector"
}
{
  "name": "gitlab-operator-cert-manager-webhook-f7d544874-td7f8",
  "sa": "gitlab-operator-cert-manager-webhook"
}
{
  "name": "gitlab-postgresql-0",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-redis-master-0",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-registry-5d694777ff-f8wgn",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-registry-5d694777ff-qrggm",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-sidekiq-all-in-1-v2-5b4bb79495-6bn27",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-toolbox-5495588b46-wxclz",
  "sa": "gitlab-app-nonroot"
}
{
  "name": "gitlab-webservice-default-5bbf4d9bfb-7jrpw",
  "sa": "gitlab-app-anyuid"
}
{
  "name": "gitlab-webservice-default-5bbf4d9bfb-wwxcp",
  "sa": "gitlab-app-anyuid"
}

Confirming capabilities dropped:

$ k get po gitlab-toolbox-6695fdbc75-wkwlk -ojson | jq '.spec.containers[0].securityContext.capabilities'
{
  "drop": [
    "KILL",
    "MKNOD",
    "SETGID",
    "SETUID"
  ]
}