Support custom annotations in shared-secrets chart
Summary
I try installing Gitlab on Kubernetes (on-prem) using helm. Cluster utilizes istio as well. Installation is successfull disabling istio injection for the namespace. Once enabled, gitlab-shared-secrets fails bootup.
Steps to reproduce
Label Namespace for failure before starting installation (commenting the label makes it work):
apiVersion: v1
kind: Namespace
metadata:
name: gitlab
labels:
istio-injection: enabled
Configuration used
helm upgrade --namespace gitlab --install gitlab gitlab/gitlab \
--timeout 600s \
--set global.hosts.domain=gitlab-nginx-ingress-controller.gitlab.svc.cluster.local \
--set certmanager-issuer.email=gitlab@example.com \
--set postgresql.install=false \
--set global.psql.host=postgres.postgresql.svc.cluster.local \
--set global.psql.database=gitlab \
--set global.psql.username=system_gitlab \
--set global.psql.password.secret=gitlab-secrets \
--set global.psql.password.key=psql_pwd
apiVersion: v1
kind: Secret
metadata:
name: gitlab-secrets
namespace: gitlab
type: Opaque
data:
psql_usr: removed
psql_pwd: removed
Further, matching (and working) PVs are created prior to helm install:
- gitaly
- minio
- postgresql (will be removed later on)
- prometheus
- redis-master
Current behavior
gitlab-shared-secrets fails during boot:
kubectl logs -f pod/gitlab-shared-secrets-1-a0a-6jzv6 -n gitlab shared-secrets
/tmp/tmp.6YdFesHabG /
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
secret/gitlab-gitlab-runner-secret created
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-runner-secret labeled
secret/gitlab-gitlab-runner-secret labeled
Generating a RSA private key
............................++++
...........................................................++++
unable to write 'random state'
writing new private key to 'certs/registry-example-com.key'
-----
secret/gitlab-registry-secret created
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-secret labeled
secret/gitlab-registry-secret labeled
Generating RSA private key, 2048 bit long modulus
...........................+++++
.........+++++
unable to write 'random state'
e is 65537 (0x010001)
Generating RSA private key, 2048 bit long modulus
.......................+++++
.......+++++
unable to write 'random state'
e is 65537 (0x010001)
secret/gitlab-rails-secret created
label "app.kubernetes.io/name" not found.
secret/gitlab-rails-secret labeled
secret/gitlab-rails-secret labeled
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
secret/gitlab-gitlab-shell-host-keys created
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-shell-host-keys labeled
secret/gitlab-gitlab-shell-host-keys labeled
secret/gitlab-gitlab-workhorse-secret created
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-workhorse-secret labeled
secret/gitlab-gitlab-workhorse-secret labeled
secret/gitlab-registry-httpsecret created
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-httpsecret labeled
secret/gitlab-registry-httpsecret labeled
I verified 10.96.0.1:443 to be reachable from gitlab namespace using an alpine container (istio-enabled).
kubectl get all -o wide -n gitlab
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/tmp-shell 2/2 Running 0 65s 192.168.4.238 kubernetes-slave <none> <none>
kubectl run --generator=run-pod/v1 tmp-shell --rm -i --tty --image nicolaka/netshoot -n gitlab -- /bin/bash
curl https://10.96.0.1:443/api --insecure
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/api\"",
"reason": "Forbidden",
"details": {
},
"code": 403
Helm running as cluster user (cleanup, once working). Since it works without istio, permission seem not to be the issue.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-pod-reader
rules:
- apiGroups: ["", "apps", "networking.k8s.io", "networking.istio.io", "extensions", "policy", "apiextensions.k8s.io", "rbac.authorization.k8s.io", "autoscaling", "batch", "metrics.k8s.io"]
resources: ["namespaces", "virtualservices", "persistentvolumes", "persistentvolumeclaims", "deployments", "services", "pods", "pods/exec", "networkpolicies", "secrets", "ingresses", "poddisruptionbudgets", "serviceaccounts", "configmaps", "customresourcedefinitions", "clusterroles", "clusterrolebindings", "roles", "rolebindings", "horizontalpodautoscalers", "statefulsets", "jobs", "services/proxy", "nodes"]
verbs: ["create", "delete", "get", "watch", "list", "patch", "update", "proxy"]
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
Expected behavior
gitlab-shared-secrets boots up and installs gitlab, having istio enabled.
Versions
- Chart: gitlab/gitlab
- Platform:
- Self-hosted: vanilla kubernetes
- Kubernetes: (
kubectl version
)- Client: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
- Server: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
- Helm: lachlanevenson/k8s-helm:latest
- Client: v.3.4.2
- Server: v.3.4.2
Relevant logs
kubectl get all -o wide -n gitlab Kubernetes-Master: Sun Dec 20 09:38:34 2020
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/gitlab-shared-secrets-1-il9-jqt6j 1/2 NotReady 0 19m 192.168.4.247 kubernetes-slave <none> <none>
NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR
job.batch/gitlab-shared-secrets-1-il9 0/1 19m 19m shared-secrets registry.gitlab.com/gitlab-org/build/cng/kubectl:1.13.12 controller-uid=30327316-19d5
-443f-853f-c8429750886e
Edited by Mitchell Nielsen