`gitlab-shell` has a lot of liveness errors in Review Apps

As can be seen here: https://console.cloud.google.com/logs/viewer?authuser=0&project=gitlab-review-apps&minLogLevel=0&expandAll=false&timestamp=2019-10-31T12:21:01.486000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeStart=2019-10-31T11:24:02.202Z&dateRangeEnd=2019-10-31T12:24:02.202Z&interval=PT1H&resource=k8s_pod&advancedFilter=resource.type%3D%22k8s_pod%22%0Aresource.labels.project_id%3D%22gitlab-review-apps%22%0Aresource.labels.location%3D%22us-central1-b%22%0Aresource.labels.cluster_name%3D%22review-apps-ee%22%0Aresource.labels.namespace_name%3D%22review-apps-ee%22%0Aresource.labels.pod_name:%22gitlab-shell%22%0AjsonPayload.message:%22Liveness%20probe%20errored%22&scrollTimestamp=2019-10-31T12:23:48.000000000Z and here (purple line):

logging_user_liveness_errors_per_component

the gitlab-shell component has a lot of liveness errors in Review Apps. We should consider increasing the timeout.

Samples

git@review-34824-lice-la0max-gitlab-shell-c8f597db9-xsjk8:/$ time /scripts/healthcheck
13

real	0m17.512s
user	0m2.617s
sys	0m0.038s
git@review-34824-lice-la0max-gitlab-shell-c8f597db9-xsjk8:/$ time /scripts/healthcheck
13

real	0m11.159s
user	0m1.654s
sys	0m0.040s
git@review-34824-lice-la0max-gitlab-shell-c8f597db9-xsjk8:/$ time /scripts/healthcheck
13

real	0m12.526s
user	0m1.721s
sys	0m0.031s
git@review-34416-subs-oq6do3-gitlab-shell-7c787fc45-kdzln:/$ time /scripts/healthcheck
14

real	0m13.102s
user	0m2.694s
sys	0m0.075s
git@review-34416-subs-oq6do3-gitlab-shell-7c787fc45-kdzln:/$ time /scripts/healthcheck
14

real	0m7.266s
user	0m1.783s
sys	0m0.015s
git@review-34416-subs-oq6do3-gitlab-shell-7c787fc45-kdzln:/$ time /scripts/healthcheck
14

real	0m5.784s
user	0m1.524s
sys	0m0.018s
git@review-34416-subs-oq6do3-gitlab-shell-7c787fc45-kdzln:/$ time /scripts/healthcheck
14

real	0m6.472s
user	0m2.008s
sys	0m0.087s
Edited Oct 31, 2019 by Rémy Coutable
Assignee Loading
Time tracking Loading