Forward Kubernetes events to the job log
Description
In the existing implementation, when employing the Kubernetes executor, the job logs the following message during the setup of the job Pod.
Waiting for pod gitlab-runners/runner-etjxlzsb-project-3830-concurrent-0ghjh2 to be running, status is Pending
Waiting for pod gitlab-runners/runner-etjxlzsb-project-3830-concurrent-0ghjh2 to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
While this message can occasionally be linked to a Kubernetes cluster issue, in specific cases, the underlying cause can be identified in the events recorded for the Pod. To enhance customer understanding of the job Pod's status, we could consider streaming all events related to the Pod directly within the job log.
Proposal
For each job:
- Start the monitoring of events associated with the pod, irrespective of their event type
- Stream back any event received to the job log
Points to consider
- Replace the existing polling mechanism for Pod Status by event watching
- Log a message/event only once. This might require that we stop logging the Pod Status multiple times
Links to related issues and merge requests / references
Edited by Romuald Atchadé