Fix flaky TestPodWatcher/deleted by syncing on fake client watch start
Root Cause
TestPodWatcher/deleted (executors/kubernetes/internal/watchers/pod_test.go) intermittently failed because of a race in client-go's fake Kubernetes clientset: cache.WaitForCacheSync only waits for an informer's initial List to be processed, not for its subsequent Watch call to be registered with the fake tracker.
A test that creates/deletes objects immediately after WaitForCacheSync returns can race past that registration: the fake tracker has no watcher yet to notify, and it does not replay missed events once one is registered, so the event is dropped permanently rather than merely delayed. Because the pod-deletion event genuinely never arrived, no amount of widening waitForError's timeout could fix the flake.
Fix
Added a newFakeKubeClientWithSyncedWatch helper that installs a PrependWatchReactor on the fake clientset and returns a waitForWatchStart function. TestPodWatcher and TestPodWatcherNoConsumer now call waitForWatchStart right after podWatcher.Start(), blocking until the fake watch has actually been registered before mutating the pod. This closes the race at its source instead of papering over it with a longer timeout.
This replaces two earlier, reverted approaches from this MR's history: widening waitForError's timeout to 5 * time.Second everywhere, and later parameterizing that timeout per-subtest. Neither addressed the actual root cause, since the missing event was never going to arrive no matter how long the test waited.
Files Modified
executors/kubernetes/internal/watchers/pod_test.go
What are the relevant issue numbers?
Closes #39640 (closed)
Originally generated by the Duo Fix Pipeline flow fixing https://gitlab.com/gitlab-org/gitlab-runner/-/pipelines/2715227615 (session); picked up and finished off draft to close out the tracked flaky-test issue.