Some jobs claim to upload artifacts but artifacts aren't actually there
As can be seen in https://gitlab.com/gitlab-org/gitlab/-/jobs/483665351, https://gitlab.com/gitlab-org/gitlab/-/jobs/483665367 (vs the normal jest
job here: https://gitlab.com/gitlab-org/gitlab/-/jobs/483665359), as well as https://gitlab.com/gitlab-org/gitlab/-/jobs/483275772.
Some artifacts aren't downloaded when they should be. For instance jest-as-if-foss
has needs: ["frontend-fixtures-as-if-foss"]
but no artifacts were downloaded here: https://gitlab.com/gitlab-org/gitlab/-/jobs/483665367.
In https://gitlab.com/gitlab-org/gitlab/-/jobs/483275772, only the artifacts from compile-assets pull-cache as-if-foss
weren't downloaded (i.e. setup-test-env
and retrieve-tests-metadata
artifacts were downloaded).
Current state of investigation
It seems that after uploading artifacts they may be missing as database entry. The files do exist on object storage and successful request to object storage is observed. However we have no error response to the Runner. It seems that the API response returns 201 Created
which occurs only after the transaction persists both archive artifact and metadata. Transaction failure should cause a 400 error as well as Sentry report which did not occur.
Another hypothesis may be that something external (perhaps ExpireBuildArtifactsWorker
) may delete the artifact but it should only delete expired ones.
from #212349 (comment 428913634):
- The presence of the transaction should at least ensure that both metadata and zip archive must be inserted, otherwise we should see errors in sentry and logs. This was not the case
- I thought the same about the
ExpireBuildArtifactsWorker
that would only remove expired artifacts which seem to be set correctly to 7 days later, looking atbuild.artifacts_expire_at
which is set after persisting the artifacts