External repository dependency fails k8s deployment
Seen in https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/2731221, and https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/2727833
Adding repo gitlab git+ssh://git@dev.gitlab.org/gitlab/charts/gitlab.git@?ref=da1b601434cfa0bb08a46d109d5437290ec56829
"gitlab" has been added to your repositories
tail: warning: --retry only effective for the initial open
tail: cannot open '/builds/gitlab-com/gl-infra/k8s-workloads/gitlab-com/tiller.log' for reading: No such file or directory
Comparing release=gitlab, chart=gitlab/gitlab
tail: '/builds/gitlab-com/gl-infra/k8s-workloads/gitlab-com/tiller.log' has appeared; following new file
[main] 2020/12/30 14:07:05 Starting Tiller v2.16.9 (tls=false)
[main] 2020/12/30 14:07:05 GRPC listening on 127.0.0.1:44134
[main] 2020/12/30 14:07:05 Probes listening on 127.0.0.1:44135
[main] 2020/12/30 14:07:05 Storage driver is Secret
[main] 2020/12/30 14:07:05 Max history per release is 20
in ./helmfile.yaml: in .helmfiles[2]: in releases/gitlab/helmfile.yaml: command "/usr/local/bin/helm" exited with non-zero status:
PATH:
/usr/local/bin/helm
ARGS:
0: helm (4 bytes)
1: tiller (6 bytes)
2: run (3 bytes)
3: kube-system (11 bytes)
4: -- (2 bytes)
5: helm (4 bytes)
6: diff (4 bytes)
7: upgrade (7 bytes)
8: --reset-values (14 bytes)
9: --allow-unreleased (18 bytes)
10: gitlab (6 bytes)
11: gitlab/gitlab (13 bytes)
12: --namespace (11 bytes)
13: gitlab (6 bytes)
14: --values (8 bytes)
15: /tmp/values882747381 (20 bytes)
16: --values (8 bytes)
17: /tmp/values750195920 (20 bytes)
18: --values (8 bytes)
19: /tmp/values920503791 (20 bytes)
20: --values (8 bytes)
21: /tmp/values277076098 (20 bytes)
22: --set (5 bytes)
23: global.deployment.annotations.app\.gitlab\.com/app=gitlab-com-gl-infra-k8s-workloads-gitlab-com (95 bytes)
24: --set (5 bytes)
25: global.deployment.annotations.app\.gitlab\.com/env=gstg-us-east1-b (66 bytes)
26: --set (5 bytes)
27: global.registry.bucket=gitlab-\{\{ .Environment.Values | getOrNil "env_prefix" | default .Environment.Name \}\}-registry (120 bytes)
28: --set (5 bytes)
29: registry.annotations.app\.gitlab\.com/app=gitlab-com-gl-infra-k8s-workloads-gitlab-com (86 bytes)
30: --set (5 bytes)
31: registry.annotations.app\.gitlab\.com/env=gstg-us-east1-b (57 bytes)
32: --detailed-exitcode (19 bytes)
33: --suppress-secrets (18 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
Debug in plugin 'helm-git': repo: ssh://git@dev.gitlab.org/gitlab/charts/gitlab.git ref: da1b601434cfa0bb08a46d109d5437290ec56829 path: file: gitlab-4.6.3.tgz sparse: 1
Debug in plugin 'helm-git': helm_repo_uri: git+ssh://git@dev.gitlab.org/gitlab/charts/gitlab.git@?ref=da1b601434cfa0bb08a46d109d5437290ec56829&sparse=1
Error: could not find : chart postgresql not found in https://charts.bitnami.com/bitnami
Error in plugin 'helm-git': Error while helm_dependency_update
Error: plugin "helm-git" exited with error
Error: plugin "diff" exited with error
Error: plugin "tiller" exited with error
COMBINED OUTPUT:
Installing Tiller v2.16.9 ...
Debug in plugin 'helm-git': repo: ssh://git@dev.gitlab.org/gitlab/charts/gitlab.git ref: da1b601434cfa0bb08a46d109d5437290ec56829 path: file: gitlab-4.6.3.tgz sparse: 1
Debug in plugin 'helm-git': helm_repo_uri: git+ssh://git@dev.gitlab.org/gitlab/charts/gitlab.git@?ref=da1b601434cfa0bb08a46d109d5437290ec56829&sparse=1
Error: could not find : chart postgresql not found in https://charts.bitnami.com/bitnami
Error in plugin 'helm-git': Error while helm_dependency_update
Error: plugin "helm-git" exited with error
Error: plugin "diff" exited with error
Error: plugin "tiller" exited with error
We depend on bitnami repository which appears to sometimes fail. We should work with Distribution to find a way to remove this dependency.
Solution
The first iteration of this work is now complete. Please see the MR for further detail. I consider this 2/3's complete for removing our dependencies. Instead of building items per job, we now build the chart for each pipeline. So we currently still suffer from a dependency issue, but at least the foot print where failure occurs is much less. Bonus, we've implemented fall backs, such that if the build of the chart does fail, we can still utilize the git repo and use the exiting method.
There are some improvements we can do, but I do not consider these pressing items.
- Clean up the feature flag - we should let it bake for a period of time
-
Improvements - Either wait for the improvement to be made to the artifact API
- Doing so will allow us to build a better pipeline that only builds artifacts once and then we can simply query for that artifact object in future pipelines
- This reduces our failure footprint even further
- Alternatively, we can leverage the use of the Container Registry instead of an artifact - a commonly used pattern with omnibus builds