Review apps cleanup is not complete

added typebug label

Two things I notice:

It seems the Helm command to retrieve the releases is missing the namespace: helm list --namespace "" --deployed --failed --date --reverse --max 25
- In HelmClient we're fetching ENV['KUBE_NAMESPACE'] but here it's empty
- I think that's because originally the cleanup task was done in the review job itself, thus kubernetes: active was there, but now that we're running the job in as a pipeline schedule, kubernetes: active isn't there anymore
When deleting environments, we should hard cleanup them right away with Helm because we're not going to stop the environment so we can just clean it up ASAP

assigned to @rymai

changed milestone to %11.4

It seems the Helm command to retrieve the releases is missing the namespace:

Wow, I see it now @rymai but I completely missed it when I was looking at that job

@marin I think we won't be able to clean up review apps if kubernetes: active isn't set, I think we need that so that Helm/Kubernetes command talk to the expected cluster, right? See https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/103897123, there are no releases listed even with the correct namespace...

I think so, kubernetes keyword in CI seems to be responsible for enabling kubernetes service which then has a set of predefined_variables added to the CI variables (One of them being KUBE_NAMESPACE).

See https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/103897123, there are no releases listed even with the correct namespace.

Reason why you are not seeing any releases there is different to using kubernetes: active. Helm's tiller is installed in a specific namespace. By default, tiller is in default namespace while our app is in the review-apps-ee namespace.

So you need to do:

helm list --short --namespace "review-apps-ee" --tiller-namespace 'review-apps-ee'

in order to list deployments. Additionally, start with max of 10 releases to delete. Tiller pod will die with 50.

added to epic &265 (closed)

@marin I was very hopeful, but it still doesn't seem to work: https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/104252434

@rymai Did we run that command exactly, so helm list --namespace "review-apps-ee" --tiller-namespace "review-apps-ee" --deployed --failed --date --reverse --max 10? If that is the command, it only lists releases :D

To actually delete, we need to feed it to delete command (like I left in the description):

helm list --short --namespace "review-apps-ee" --tiller-namespace 'review-apps-ee' --failed --date --max 10 | xargs -L1 helm delete --purge --tiller-namespace 'review-apps-ee'

Did we run that command exactly, so helm list --namespace "review-apps-ee" --tiller-namespace "review-apps-ee" --deployed --failed --date --reverse --max 10? If that is the command, it only lists releases :D

@marin Aha, yes of course that only lists releases, fortunately I'm aware of that , but I think this is probably the first problem: it seems we don't retrieve any releases with this command because for each release in the list, a message should be displayed: either we don't touch the review app, or we delete it, but here (https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/104252434), no messages is printed at all, which probably means that the list is empty... I'm going to debug that in a GKE console!

@rymai haha, sorry I had to make sure ;)

I think I may have found something, helm list returns something like

NAME                            REVISION        UPDATED                         STATUS          CHART           APP VERSION     NAMESPACE
review-6717-add-s-zuf05x        1               Thu Oct  4 17:42:34 2018        FAILED          gitlab-1.1.3    master          review-apps-ee
review-7846-fix-s-261vd6        1               Thu Oct  4 17:33:29 2018        FAILED          gitlab-1.1.3    master          review-apps-ee
review-7867-snowp-lzo3iy        1               Thu Oct  4 17:22:14 2018        DEPLOYED        gitlab-1.1.3    master          review-apps-ee
review-rename-geo-o4a780        1               Thu Oct  4 17:14:57 2018        DEPLOYED        gitlab-1.1.3    master          review-apps-ee
review-5781-opera-0k93fx        1               Thu Oct  4 17:06:15 2018        FAILED          gitlab-1.1.3    master          review-apps-ee
review-6709-group-2pzeec        1               Thu Oct  4 16:36:59 2018        FAILED          gitlab-1.1.3    master          review-apps-ee
review-ce-to-ee-2-l554mn        1               Thu Oct  4 16:27:02 2018        FAILED          gitlab-1.1.3    master          review-apps-ee
review-epics-e2e-m690eb         1               Thu Oct  4 16:08:26 2018        DEPLOYED        gitlab-1.1.3    master          review-apps-ee
review-7126-admin-06fae2        1               Thu Oct  4 15:56:35 2018        FAILED          gitlab-1.1.3    master          review-apps-ee
review-6983-promo-xyou11        1               Thu Oct  4 15:15:34 2018        FAILED          gitlab-1.1.3    master          review-apps-ee

But in my HelmClient, I define a release as Release = Struct.new(:name, :revision, :last_update, :status, :chart, :namespace) and then I populate releases with Release.new(*line.split(/\t/).map(&:strip)), so the APP VERSION data ends up in the namespace attribute... But even if this is an issue, I should still have a list of releases in scripts/automated_cleanup.rb. The investigation continues...

I think I'm just going to output the list as JSON and parse it, that will result in more predictable results...

Hmm, even after fetching & parsing the JSON output, it's obvious we don't get any releases:

Running command: `helm list --namespace "review-apps-ee" --tiller-namespace "review-apps-ee" --output json --deployed --failed --date --reverse --max 10 --debug`
Ignoring this JSON parsing error: 751: unexpected token at ''

But if I run the same command in the GKE console I get results:

remy@cloudshell:~ (gitlab-review-apps)$ helm list --namespace "review-apps-ee" --tiller-namespace "review-apps-ee" --output json --deployed --failed --date --reverse --max 10 --debug
[debug] Created tunnel using local port: '45621'
[debug] SERVER: "127.0.0.1:45621"
{"Next":"review-6983-promo-xyou11","Releases":[{"Name":"review-6717-add-s-zuf05x","Revision":1,"Updated":"Thu Oct  4 18:04:32 2018","Status":"FAILED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name":"review-5940-manua-1tr3jh","Revision":1,"Updated":"Thu Oct  4 18:01:59 2018","Status":"FAILED
","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name":"review-qa-60-reor-1mugd1","Revision":1,"Updated":"Thu Oct  4 17:52:31 2018","Status":"FAILED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name":"review-7846-fix-s-261vd6","Revision":1,"Updated":"Thu Oct  4 1
7:33:29 2018","Status":"FAILED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name":"review-7867-snowp-lzo3iy","Revision":1,"Updated":"Thu Oct  4 17:22:14 2018","Status":"DEPLOYED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name":"review-rename-geo-o4a780","Rev
ision":1,"Updated":"Thu Oct  4 17:14:57 2018","Status":"DEPLOYED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name":"review-6709-group-2pzeec","Revision":1,"Updated":"Thu Oct  4 16:36:59 2018","Status":"FAILED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name"
:"review-ce-to-ee-2-l554mn","Revision":1,"Updated":"Thu Oct  4 16:27:02 2018","Status":"FAILED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"},{"Name":"review-epics-e2e-m690eb","Revision":1,"Updated":"Thu Oct  4 16:08:26 2018","Status":"DEPLOYED","Chart":"gitlab-1.1.3","AppVersion":"master","Names
pace":"review-apps-ee"},{"Name":"review-7126-admin-06fae2","Revision":1,"Updated":"Thu Oct  4 15:56:35 2018","Status":"FAILED","Chart":"gitlab-1.1.3","AppVersion":"master","Namespace":"review-apps-ee"}]}

@rymai Have you tried just running the command with system(), avoiding the ruby wrapper?

mentioned in issue #52328 (closed)

added severity1 label

changed milestone to %11.5

@marin Same error in https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/106264017 but now a new error is shown, which might explain the failure: Error: Get http://localhost:8080/api/v1/namespaces/review-apps-ee/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp [::1]:8080: connect: connection refused.

Running command: `helm list --namespace "review-apps-ee" --tiller-namespace "review-apps-ee" --output json --deployed --failed --date --reverse --max 10 --debug`
Error: Get http://localhost:8080/api/v1/namespaces/review-apps-ee/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp [::1]:8080: connect: connection refused
Ignoring this JSON parsing error: 751: unexpected token at ''

@marin Ok, I found the problem, the job was missing

  environment:
    name: review/auto-cleanup
    action: stop

Without that, the Kubernetes integration isn't "enabled" in the job. I've hard-coded the environment here because we don't want to let GitLab think that we stopped a legit review app in the scheduled pipeline (which is what would happen if we'd use review/$CI_COMMIT_REF_NAME).

Now, I get other Helm issues, but I think I can manage to solve them by fetching less releases at a time, and/or retrying the commands... :)

Ok, that is a bit unexpected :O I thought that only kuberentes: active "enables" the integration...

Some quick digging this morning. The tiller pods for review-apps-ee are getting evicted periodically due to low memory on a node. Due to the load on the cluster, it can take a few minutes to respawn, and helm commands fail during this time.

It looks like helm does support multiple replicas. But based on https://github.com/helm/helm/pull/3464 I'm not convinced it is safe to use in this instance.

mentioned in issue #49571 (moved)

mentioned in issue #53713 (closed)

mentioned in issue #55323 (closed)

Review apps cleanup is not complete

Designs

Child items ...

Activity