Skip to content

[K8S Dogfooding] Cluster warning events are printed out of order, after the pod's termination

In https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/9393107586 on line 14 we can see the warning:

Warning   FailedScheduling   0/5 nodes are available: 1 node(s) had untolerated taint {ToBeDeletedByClusterAutoscaler: 1741810825}, 2 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 2 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.

This is OK, it means that we need the cluster autoscaler to provision a new node before the pod can be scheduled.

This particular build pod was terminated with OOM later. After it was requested to be terminated the warnings were printed again on lines 377 and onwards.

This is either an out of order printing or we are duplicating it it feels like.

We have the FF_PRINT_POD_EVENTS=true FF set.

Edited by Georgi N. Georgiev | GitLab