fleet-cleanup-gitrepo-jobs test KO in mgmt-cronjob-testing CI test job
In our downstream CI based on sylva-core release-1.7 branch, we observed a failure of the "mgmt-cronjob-testing" CI test job that relies on the .gitlab/ci/scripts/cronjob-test.sh script coming from sylva 1.7.8 tag.
In the "mgmt-cronjob-testing" CI test job output, we can see:
$ .gitlab/ci/scripts/cronjob-test.sh
Creating job fleet-cleanup-gitrepo-jobs-test-58cb525f from cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system...
...
job.batch/fleet-cleanup-gitrepo-jobs-test-58cb525f created
Waiting for job fleet-cleanup-gitrepo-jobs-test-58cb525f to complete...
...
Error from server (NotFound): jobs.batch "fleet-cleanup-gitrepo-jobs-test-58cb525f" not found
KO: Job fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system did not complete successfully within timeout.
Dumping fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system
...
Error from server (NotFound): jobs.batch "fleet-cleanup-gitrepo-jobs-test-58cb525f" not found
error: error from server (NotFound): jobs.batch "fleet-cleanup-gitrepo-jobs-test-58cb525f" not found in namespace "cattle-fleet-system"
...
CronJob Test Report - Wed Oct 28 12:30:40 UTC 2026
------------------------------------
OK: Job keycloak-postgresql-healer-test-58cb525f for cronjob keycloak-postgresql-healer in namespace keycloak completed successfully.
KO: Job fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system did not complete successfully within timeout.
OK: Job descheduler-test-58cb525f for cronjob descheduler in namespace kube-system completed successfully.
OK: Job longhorn-instance-manager-cleanup-test-58cb525f for cronjob longhorn-instance-manager-cleanup in namespace longhorn-system completed successfully.
OK: Job keycloak-users-configmap-test-58cb525f for cronjob keycloak-users-configmap in namespace keycloak completed successfully.
OK: Job rke2-machineconfig-cleanup-cronjob-test-58cb525f for cronjob rke2-machineconfig-cleanup-cronjob in namespace fleet-default completed successfully.
OK: Job rke2-machineconfig-cleanup-cronjob-test-58cb525f for cronjob rke2-machineconfig-cleanup-cronjob in namespace fleet-local completed successfully.
KO: Job fleet-cleanup-gitrepo-jobs-test-58cb525f for cronjob fleet-cleanup-gitrepo-jobs in namespace cattle-fleet-system did not complete successfully within timeout.
Some jobs failedIn the job artifacts, there are no info related to fleet-cleanup-gitrepo-jobs (no logs, no job yaml definition).
When we run the job manually, the job disappears quickly:
$ kubectl create job --from=cronjob/fleet-cleanup-gitrepo-jobs fleet-cleanup-gitrepo-jobs-test-gl -n cattle-fleet-system
job.batch/fleet-cleanup-gitrepo-jobs-test-gl created
$ kubectl get job -n cattle-fleet-system
No resources found in cattle-fleet-system namespace.
$ kubectl get job -A|grep fleet
fleet-default rke2-machineconfig-cleanup-cronjob-29885765 Complete 1/1 17s 15hWhen we look at the events in the cattle-fleet-system namespace, we can see that the job is created, but that it is deleted by cronjob/fleet-cleanup-gitrepo-jobs object:
ubuntu@bm-bootstrap-vm-ocalif3:~$ k get events -n cattle-fleet-system
LAST SEEN TYPE REASON OBJECT MESSAGE
47m Normal Scheduled pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq Successfully assigned cattle-fleet-system/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq to management-cluster-sr250-3
47m Normal Pulled pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq Container image "registry.rancher.com/rancher/fleet:v0.13.15" already present on machine and can be accessed by the pod
47m Normal Created pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq Container created
46m Normal Started pod/fleet-cleanup-gitrepo-jobs-test-gl-l5wdq Container started
47m Normal SuccessfulCreate job/fleet-cleanup-gitrepo-jobs-test-gl Created pod: fleet-cleanup-gitrepo-jobs-test-gl-l5wdq
46m Normal Completed job/fleet-cleanup-gitrepo-jobs-test-gl Job completed
46m Warning UnexpectedJob cronjob/fleet-cleanup-gitrepo-jobs Saw a job that the controller did not create or forgot: fleet-cleanup-gitrepo-jobs-test-gl
46m Normal SuccessfulDelete cronjob/fleet-cleanup-gitrepo-jobs Deleted jobSo in the CI job, it seems there is a race condition between the cronjob-test.sh script trying to dump the job logs and the cronjob/fleet-cleanup-gitrepo-jobs object trying to delete the job it did not create.