Maven sha1 file race condition (404 error)
Summary
We see intermittent 404 errors on Maven .sha1
file uploads. This is visible in the logs: https://log.gprd.gitlab.net/goto/4f214470-d62f-11ec-aade-19e9974a7229 with about 200 errors occuring in the last week when this issue was written.
Looking at an individual package upload, we see the file is successfully uploaded and then the sha1
file is rejected. The sha1
is unique in that we look up the matching file that was uploaded just before it. What we are seeing is the sha1
upload reaches this point before the other file has been persisted in the database, returning a 404. Or, more likely, we are seeing a race condition due to database replication lag. The first file writes to a primary, and the request for the sha1
file reads from a secondary that has not yet received the new record.
Steps to reproduce
Example Project
What is the current bug behavior?
Maven package uploads fail with a 404 error on the .sha1
package file
What is the expected correct behavior?
Maven package uploads are successful
Relevant logs and/or screenshots
Output of checks
This bug happens on GitLab.com
Possible fixes
Based on the idea of replication lag, a potential solution is to force the package finder to read from the primary:
::Gitlab::Database::LoadBalancing::Session.current.use_primary do
package_file = ::Packages::PackageFileFinder
.new(package, file_name).execute!
end
Workarounds
Retrying the package upload until it succeeds.