Elasticstack post-sync delete stuck
https://gitlab.com/gitlab-org/cluster-integration/cluster-applications/-/jobs/436781972
Deleting elastic-stack
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
elastic-stack 1 Thu Feb 13 20:28:55 2020 DEPLOYED elastic-stack-1.8.0 6 gitlab-managed-apps
helmfile.yaml: basePath=.
release "elastic-stack" deleted
If I query the resources I see:
$ kc get pvc -n gitlab-managed-apps
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-elastic-stack-elasticsearch-data-0 Terminating pvc-75fa391c-4e9f-11ea-bbbb-42010a8a0080 30Gi RWO standard 65m
data-elastic-stack-elasticsearch-data-1 Terminating pvc-be688572-4e9f-11ea-bbbb-42010a8a0080 30Gi RWO standard 63m
data-elastic-stack-elasticsearch-master-0 Terminating pvc-75d49d7b-4e9f-11ea-bbbb-42010a8a0080 4Gi RWO standard 65m
data-elastic-stack-elasticsearch-master-1 Terminating pvc-b16ca0d4-4e9f-11ea-bbbb-42010a8a0080 4Gi RWO standard 64m
data-sentry-sentry-postgresql-0 Bound pvc-ad3a967e-4e9e-11ea-bbbb-42010a8a0080 8Gi RWO standard 71m
hub-db-dir Bound pvc-8c377723-4e9f-11ea-bbbb-42010a8a0080 1Gi RWO standard 65m
prometheus-alertmanager Bound pvc-5627fad5-4ea0-11ea-bbbb-42010a8a0080 2Gi RWO standard 59m
prometheus-prometheus-server Bound pvc-9202de79-4e9f-11ea-bbbb-42010a8a0080 8Gi RWO standard 64m
redis-data-sentry-sentry-redis-master-0 Bound pvc-ad757422-4e9e-11ea-bbbb-42010a8a0080 8Gi RWO standard 71m
redis-data-sentry-sentry-redis-slave-0 Bound pvc-ad65facb-4e9e-11ea-bbbb-42010a8a0080 8Gi RWO standard 71m
redis-data-sentry-sentry-redis-slave-1 Bound pvc-c09d0514-4e9e-11ea-bbbb-42010a8a0080 8Gi RWO standard 70m
$ kc describe pvc -n gitlab-managed-apps data-elastic-stack-elasticsearch-data-0
Name: data-elastic-stack-elasticsearch-data-0
Namespace: gitlab-managed-apps
StorageClass: standard
Status: Terminating (lasts <invalid>)
Volume: pvc-75fa391c-4e9f-11ea-bbbb-42010a8a0080
Labels: app=elasticsearch
component=data
release=elastic-stack
role=data
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 30Gi
Access Modes: RWO
VolumeMode: Filesystem
Events: <none>
Mounted By: elastic-stack-elasticsearch-data-0
This problem is also described in https://stackoverflow.com/questions/51358856/kubernetes-cant-delete-persistentvolumeclaim-pvc
/cc @Alexand