Dynamic provisioning of volumes not possible on IBM cloud

Summary

We tried to create install the helm chart on IBM Kubernetes service (IKS), however we were not able to set up the required volumes dynamically.

The reason is that the helm chart does not allow to provide labels and annotations for persistent volume claims (PVC) as required per IBM documentation (see https://console.bluemix.net/docs/containers/cs_storage_basics.html#kube_concepts). For us it is important to be able to supply the volume.beta.kubernetes.io/storage-class annotation and the label billingType.

Sample IKS pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: ado-dev
  name: gitlab-ado-dev-minio
  annotations:
    volume.beta.kubernetes.io/storage-class: ibmc-file-silver
  labels:
    app: minio
    chart: minio-0.4.3
    heritage: Tiller
    release: gitlab-ado-dev
    billingType: hourly
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi

In addition, when the PVCs are bound, there are access right issues to write to these volumes that need to be fixed using an init container (see https://console.bluemix.net/docs/containers/cs_troubleshoot_storage.html).

Steps to reproduce

(Please provide the steps to reproduce the issue)

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

(Paste sanitized configuration here)

Current behavior

  • I cannot provide annotations for persistent volumes created by the gitlab helm chart.
  • I cannot define custom labels for persistent volumes created by the gitlab helm chart.
  • Dynamic provisioning fails due to invalid access rights, as the runAsUser has no write access to dynamically provisioned storage.

Example of minio persistent volume:

{{- if and .Values.global.minio.enabled .Values.persistence.enabled }}
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: {{ template "minio.fullname" . }}
  namespace: {{ $.Release.Namespace }}
  labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
spec:
  accessModes:
    - {{ .Values.persistence.accessMode | quote }}
  resources:
    requests:
      storage: {{ .Values.persistence.size | quote }}
{{- if .Values.persistence.volumeName }}
  volumeName: {{ .Values.persistence.volumeName }}
{{- end }}
{{- if .Values.persistence.storageClass }}
{{- if (eq "-" .Values.persistence.storageClass) }}
  storageClassName: ""
{{- else }}
  storageClassName: "{{ .Values.persistence.storageClass }}"
{{- end -}}
{{- end }}
  selector:
{{- if .Values.persistence.matchLabels }}
    matchLabels:
{{ toYaml .Values.persistence.matchLabels | indent 6 }}
{{- end -}}
{{- if .Values.persistence.matchExpressions }}
    matchExpressions:
{{ toYaml .Values.persistence.matchExpressions | indent 6 }}
{{- end -}}
{{- end }}

Expected behavior

  • Allow to specify custom annotations and labels for the PersistentVolumeClaim for minio, redis and gitlay
  • Set up the access rights for the runAsUser using an init container.

Example of minio persistent volume:

{{- if and .Values.global.minio.enabled .Values.persistence.enabled }}
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: {{ template "minio.fullname" . }}
  namespace: {{ $.Release.Namespace }}
{{- if .Values.persistence.pvc.annotations }}
  annotations:
{{ toYaml .Values.persistence.pvc.annotations | indent 4 }}
{{- end }}
  labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
{{- if .Values.persistence.pvc.labels }}
{{ toYaml .Values.persistence.pvc.labels | indent 4 }}
{{- end }}
spec:
  accessModes:
    - {{ .Values.persistence.accessMode | quote }}
  resources:
    requests:
      storage: {{ .Values.persistence.size | quote }}
{{- if .Values.persistence.volumeName }}
  volumeName: {{ .Values.persistence.volumeName }}
{{- end }}
{{- if .Values.persistence.storageClass }}
{{- if (eq "-" .Values.persistence.storageClass) }}
  storageClassName: ""
{{- else }}
  storageClassName: "{{ .Values.persistence.storageClass }}"
{{- end -}}
{{- end }}
  selector:
{{- if .Values.persistence.matchLabels }}
    matchLabels:
{{ toYaml .Values.persistence.matchLabels | indent 6 }}
{{- end -}}
{{- if .Values.persistence.matchExpressions }}
    matchExpressions:
{{ toYaml .Values.persistence.matchExpressions | indent 6 }}
{{- end -}}
{{- end }}

Example container section of the minio deployment:

      initContainers:
        - name: fix-storage-permissions
          image: {{ .Values.init.image }}:{{ .Values.init.tag}}
          command: ["/bin/sh", "-c", "chown 1000:1000 {{ .Values.mountPath }};"]
          volumeMounts:
          - name: export
            mountPath: {{ .Values.mountPath }}
          securityContext:
            runAsUser: 0
            fsGroup: 0

Versions

  • Chart: 1-2-stable
  • Platform:
    • Cloud: IBM Cloud Kubernetes Service
  • Kubernetes: (kubectl version)
    • Client: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-08T16:31:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
    • Server: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3+IKS", GitCommit:"ada755932025320d0c249486935c2aa31eea55d1", GitTreeState:"clean", BuildDate:"2018-09-27T20:52:21Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version)
    • Client: &version.Version{SemVer:"v2.7.3+icp", GitCommit:"27442e4cfd324d8f82f935fe0b7b492994d4c289", GitTreeState:"dirty"}
    • Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)

Edited by 🤖 GitLab Bot 🤖