fleet-cleanup-gitrepo-jobs test KO in mgmt-cronjob-testing CI test job

In our downstream CI based on sylva-core release-1.7 branch, we observed a failure of the "mgmt-cronjob-testing" CI test job that relies on the .gitlab/ci/scripts/cronjob-test.sh script coming from sylva 1.7.8 tag.

In the "mgmt-cronjob-testing" CI test job output, we can see:

$ .gitlab/ci/scripts/cronjob-test.sh
Creating job fleet-cleanup-gitrepo-jobs-test-58cb525f from cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system...
...
job.batch/fleet-cleanup-gitrepo-jobs-test-58cb525f created
Waiting for job fleet-cleanup-gitrepo-jobs-test-58cb525f to complete...
...
Error from server (NotFound): jobs.batch "fleet-cleanup-gitrepo-jobs-test-58cb525f" not found
KO: Job fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system did not complete successfully within timeout.
Dumping fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system
...
Error from server (NotFound): jobs.batch "fleet-cleanup-gitrepo-jobs-test-58cb525f" not found
error: error from server (NotFound): jobs.batch "fleet-cleanup-gitrepo-jobs-test-58cb525f" not found in namespace "cattle-fleet-system"
...
CronJob Test Report - Wed Oct 28 12:30:40 UTC 2026
------------------------------------
OK: Job keycloak-postgresql-healer-test-58cb525f for cronjob keycloak-postgresql-healer in namespace keycloak completed successfully.
KO: Job fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system did not complete successfully within timeout.
OK: Job descheduler-test-58cb525f for cronjob descheduler in namespace kube-system completed successfully.
OK: Job longhorn-instance-manager-cleanup-test-58cb525f for cronjob longhorn-instance-manager-cleanup in namespace longhorn-system completed successfully.
OK: Job keycloak-users-configmap-test-58cb525f for cronjob keycloak-users-configmap in namespace keycloak completed successfully.
OK: Job rke2-machineconfig-cleanup-cronjob-test-58cb525f for cronjob rke2-machineconfig-cleanup-cronjob in namespace fleet-default completed successfully.
OK: Job rke2-machineconfig-cleanup-cronjob-test-58cb525f for cronjob rke2-machineconfig-cleanup-cronjob in namespace fleet-local completed successfully.
KO: Job fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system did not complete successfully within timeout.
Some jobs failed

In the job artifacts, there are no info related to fleet-cleanup-gitrepo-jobs (no logs, no job yaml definition).

When we run the job manually, the job disappears quickly:

$ kubectl create job --from=cronjob/fleet-cleanup-gitrepo-jobs fleet-cleanup-gitrepo-jobs-test-gl -n cattle-fleet-system
job.batch/fleet-cleanup-gitrepo-jobs-test-gl created

$ kubectl get job -n cattle-fleet-system
No resources found in cattle-fleet-system namespace.

$ kubectl get job -A|grep fleet
fleet-default     rke2-machineconfig-cleanup-cronjob-29885765   Complete   1/1           17s        15h

When we look at the events in the cattle-fleet-system namespace, we can see that the job is created, but that it is deleted by cronjob/fleet-cleanup-gitrepo-jobs object:

ubuntu@bm-bootstrap-vm-ocalif3:~$ k get events -n cattle-fleet-system
LAST SEEN   TYPE      REASON             OBJECT                                         MESSAGE
47m         Normal    Scheduled          pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq   Successfully assigned cattle-fleet-system/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq to management-cluster-sr250-3
47m         Normal    Pulled             pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq   Container image "registry.rancher.com/rancher/fleet:v0.13.15" already present on machine and can be accessed by the pod
47m         Normal    Created            pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq   Container created
46m         Normal    Started            pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq   Container started
47m         Normal    SuccessfulCreate   job/fleet-cleanup-gitrepo-jobs-test-gl         Created pod: fleet-cleanup-gitrepo-jobs-test-gl-l5wdq
46m         Normal    Completed          job/fleet-cleanup-gitrepo-jobs-test-gl         Job completed
46m         Warning   UnexpectedJob      cronjob/fleet-cleanup-gitrepo-jobs             Saw a job that the controller did not create or forgot: fleet-cleanup-gitrepo-jobs-test-gl
46m         Normal    SuccessfulDelete   cronjob/fleet-cleanup-gitrepo-jobs             Deleted job

So in the CI job, it seems there is a race condition between the cronjob-test.sh script trying to dump the job logs and the cronjob/fleet-cleanup-gitrepo-jobs object trying to delete the job it did not create.