Prevent freezing, when metadata artifact file has size 0
Description
We have noticed that our frontend workers and Sidekiq (for exposed artefacts) hang when the metadata.gz (artefact) file in the database has a size of 0.
How to create this issue scenario
-
Make sure that object storage is enabled, see https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/object_storage.md
-
Create a pipeline like this:
# .gitlab-ci.yml image: fedora:latest stages: - setup artifacts-test: before_script: [] stage: setup script: - echo "Test" > test.gz artifacts: paths: - test.gz expose_as: 'artifact 1'
-
Take the
metadata.gz
file from the pipeline in the database and just assign it to 0. (This happens sometimes. The file get's wrongly created bygitlab-workhorse
. We still don't know why, but that is another issue.)#rails c artifact = Ci::JobArtifact.where(file_type: "metadata").last() artifact.size = 0 artifact.save
-
Click on the browse button in your pipeline on that artifact
-
Notice how GitLab suddenly freezes and waits up to TIMEOUT (60 sec) When the artifact is exposed this also causes our sidekiq workers to get into a blocking mode, where they retry to expose the artifact, but get stuck.
Analysis
To compress the artifact, Zlib:GzipReader
is used at this location. As far as I understand, Zlib:GzipReader
in ruby is directly implemented in zlib. Therefore, you don't see exactly what happens behind the scenes in the new
call but the description says:
Zlib::GzipReader.new(io, options = {}) Creates a GzipReader object associated with io. The GzipReader object reads gzipped data from io, and parses/decompresses it. The io must have a read method that behaves same as the IO#read. The options hash may be used to set the encoding of the data. :external_encoding, :internal_encoding and :encoding may be set as in IO::new. If the gzip file header is incorrect, raises an Zlib::GzipFile::Error exception. (here is the link to it)
Also see this for IO read
After the call we now land in http_io.rb
in the read
method (here).
I have now noticed that we never read any data because eof?
is always true
because size == 0
. So we never get into this loop. Therefor out
always ends up as an empty string and this causes us to stay in the infinite loop for whatever reason - read
is constantly being called again.
Question
Does somebody understand why we stay in the infinite loop in read
when out
is an empty string?
How to fix?
- Should we check
size == 0
and return a manual created emptygz
? (I have tried this but have only received exceptions so far.) - Should we raise an error if
size == 0
before we even try to compress? - Other
Since I think that I simply lack experience here and not much is missing to solve this problem, I am so cheeky and ping some backend engineers
with whom I had to do in the course: @mayra-cabrera @mwoolf @bala.kumar @Andysoiron