[CI] Functional tests failing in 4.6 cluster due to controller reconciling in multiple namespaces

Summary

We noticed functional tests were failing in the 4.6 cluster only, and it appeared that GitLab Controllers were reconciling objects outside their own namespace (sometimes for multiple other namespaces).

Details

At first we noticed that all active namespaces seem to be reconciling objects for n065cc444, which is !186 (merged). We saw that MR's cleanup job ran (link), but the namespace and all resources were still there. Noting that now there are two cleanup jobs (one for each namespace), so we need to rebase that MR to pick up changes.

NS ncdc1972b: !182 (merged)

• No pods other than controller
• Reconciling objects for n065cc444

NS n998cf192: !185 (merged)

• No pods other than controller
• Reconciling objects for n065cc444

NS n065cc444: !186 (merged)

• All expected pods, and lots of shared secrets pods
• Reconciling objects for its own namespace

NS n122b9517:

• All expected pods, and lots of shared secrets pods
• Reconciling objects for n065cc444

NS n6028aeae: !181 (merged)

• All expected pods and lots of shared secrets pods
• Reconciling objects for n065cc444

How to replicate the issue

See related comment