GitLab 18 - CI Job logs are disappearing after a short while after they have run.
Since we upgraded multiple instances to gitlab 18 we noticed that there are severe bugs on the CI side of things.
All logs are gone (at least in the UI, I guess they are still available in the artifacts). This happens shortly after the job finishes running, already during they're running the logs behave weird though. They "flicker" and randomly show different states. It looks like 2 jobs would be showing instead of only one job. But if we look at the actual spawned jobs, they're running fine.
This ends up with ultimately ALL logs being gone and unavailable over the UI. Even jobs that ran before the upgrade.
The behavior is everywhere the same:
The /trace call returns 204 most of the time, and then sometimes data returns. And while a job is running, it returns randomly some other data (which is the cause of the flickers). The backend doesn't seem to throw any error, at least I haven't seen any.
This error appears across multiple different s3 object storages from what we were able to verify due to different installations we maintain, so this is not a small and isolated issue.