[Flaky] spec/features/backups_spec.rb - Toolbox command terminated with exit code 137
Summary
Eventual failure of the backup spec where the toolbox pod.
Job example: https://gitlab.com/gitlab-org/charts/gitlab/-/jobs/6854142019
Versions
- Chart: 7.11.1 (master at the time)
- Platform:
- Cloud: GKE
- Kubernetes: 1.29
- Client:
- Server:
- Helm: (
helm version
)- Client:
- Server:
Relevant logs
Failures:
1) Restoring a backup Backups Should be able to backup an identical tar
Failure/Error: expect(status.success?).to be(true), "Error backing up instance: #{stdout}"
Error backing up instance: Defaulted container "toolbox" out of: toolbox, certificates (init), configure (init)
Unable to use a TTY - input is not a terminal or the right kind of file
command terminated with exit code 137
# ./spec/features/backups_spec.rb:106:in `block (3 levels) in <top (required)>'
From https://refine.dev/blog/kubernetes-exit-code-137/
Exit code 137 is a signal that occurs when a container's memory exceeds the memory limit provided in the pod specification. When a container consumes too much memory, Kubernetes kills it to protect it from consuming too many resources on the node. This ensures that other containers on the same node are able to keep running normally.
We don't explicitly set limit
on memory. So I wonder what is the limit computed to this pod.
We don't seem to have a ResourceQuota for the helm-charts-win
namespace:
➜ ~ kc get resourcequotas --all-namespaces
NAMESPACE NAME AGE REQUEST LIMIT
gke-managed-system gcp-critical-pods 32d pods: 0/1G
kube-system gcp-critical-pods 32d pods: 48/1G
Is it defaulting to kube-system
them?
Describing the pod does not print the limits:
kubectl get pods rvw-a2e7ffc4-toolbox-5c9644b8b6-4lz2m -n helm-charts-win -o jsonpath='{range .spec.containers[*]}{"Container Name: "}{.name}{"\n"}{"Requests:"}{.resources.requests}{"\n"}{"Limits:"}{.resources.limits}{"\n"}{end}'
Container Name: toolbox
Requests:{"cpu":"50m","memory":"350M"}
Limits: