DoS via malicious test report artifacts

HackerOne report #1774688 by luryus on 2022-11-15, assigned to @ngeorge1:

Report

Summary

By uploading a crafted gzip file (basically a zip bomb) as a JUnit test report artifact to a CI job, an attacker can cause DoS in both Sidekiq (background jobs) and Puma (web server). The attack works by causing excessive memory use and OOMKills in the services. After setting up the attack in a public project, the Puma DoS can be triggered and sustained by unauthenticated users by making simple API requests.

In smaller Gitlab instances, this attack can prevent normal usage as Puma gets stuck handling the attack requests (100% cpu until crashing due to OOMKill). At least on my own test instance (small-ish VM running Gitlab in Docker), while the attack is ongoing, Gitlab responds to requests very slowly or not at all.

The attack can not be mitigated by rate limits:

Puma DoS: The amount of API requests to make to sustain the attack is relatively low; a couple of requests a second is enough with my instance. This is very, very easy to achieve especially as no authentication is required.

Sidekiq DoS: Because the attacker only needs to upload a single, small (a few megabyte) file to achieve this, it's difficult or impossible to mitigate this with rate limits. This attack can be repeated as frequently as the attacker can create pipelines, and therefore the attacker can continuously cause crashes.

I used Gitlab's CVSS calculator to determine the "high" severity here (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:N/I:N/A:H).

Steps to reproduce

Create a payload

A payload file is included in the attached sample project, but here's how to generate one:

touch report.json  
fallocate -l 12GiB report.json  
gzip report.json  
###  After gzip finishes, the report.json.gz file is the payload

Sidekiq DoS

In your own Gitlab instance, import the attached project as a publicly-accessible project
- Side note: many Gitlab instances (e.g. Gitlab.com) allow users to sign up themselves and create public projects, so in those cases the attacker does not need any prior access to the instance
Go to the project CI settings and configure a Docker-based Gitlab runner for it.
Monitor the sidekiq processes in the Gitlab instance with, for example, these tools:
- htop for general system memory usage and processes
- tail -f path/to/sidekiq/logs for monitoring sidekiq working (exact log file path depends on the installation)
- dmesg -T -w for kernel logs (OOMKill logs end up here)
In the imported Gitlab project, go to CI/CD -> Pipelines, click "Run pipeline", and create a new pipeline for the main branch
The CI will upload the payload Gzip file. Sidekiq will crash soon after, when it tries to read the large 12 gigabyte file from the gzip into memory. The memory use rising and Sidekiq then getting killed can be observed with the tools listed above.

Puma / Gitlab API DoS

Run the Sidekiq DoS process described above at least once.
Find the following details:
- Project ID of the imported sample project (shown in the project main page)
- Pipeline ID of the pipeline created in step 4 above
Monitor the sidekiq processes in the Gitlab instance with, for example, these tools:
- htop for general system memory usage and processes
- dmesg -T -w for kernel logs (OOMKill logs end up here)
Run the following command that makes unauthenticated API requests in a slow-ish loop. Replace the placeholders with the IDs found in step 2. The attack loop can be stopped with Ctrl+C.
```
while true; do sleep 0.5 ; curl -m 0.1 'http://<GITLAB_ADDRESS>/api/v4/projects/<PROJECT_ID>/pipelines/<PIPELINE_ID>/test_report' ; done  
```
- curl is given the -m flag to prevent it from waiting for the response. Gitlab will still try to handle the request fully even though curl disconnects.
While the attack is ongoing:
- Observe high CPU usage and puma OOMKills in htop and logs
- Try to use Gitlab normally, observe that it responds very slowly or not at all.

Impact

Sidekiq DoS: An attacker can get a sidekiq worker OOMKilled by a simple CI job file upload. This will interrupt any background jobs running on that particular worker. Because the attack is very simple, the attacker can do this often to continuously cause crashes. This can affect any user in the Gitlab instance because much of Gitlab's functionality relies on sidekiq jobs. For instance, this may cause a CI pipeline to fail and be left in a "pending" state for a long time, if a background job for that pipeline was running when sidekiq crashed.
Puma DoS: An attacker can make Puma try to decompress a previously uploaded large zip file, causing it to use excessive CPU resources and memory and then get OOMKilled. Constant crashes and high resource usage severely limit Puma's ability to serve normal user requests. As a result, Gitlab essentially may become unusable for all users.

Examples

Example repro project is attached.

Also attached is a video where I demonstrate the issue in my own small Gitlab installation.

What is the current bug behavior?

Gitlab uses the Gitlab::Ci::Build::Artifacts::Adapters::GzipStream class to unpack report artifacts (test reports, coverage reports etc.). This class is used from multiple places (through multiple abstraction levels): some Sidekiq background jobs (e.g. Ci::BuildFinishedWorker) and directly with some API requests (e.g. the example API request used above).

GzipStream does not limit the amount of data read from the zip file; it just tries to buffer the entire contents in memory (by calling gz.read with no length limit). Note that this vulnerability affects all reports and other artifacts that use GzipStream; I was able to crash Sidekiq with coverage reports, too.

An attacker can gzip a large empty file to make a small gzip file expand to a very large amount of data. In the example project I used 12GiB files because that's the memory limit in Gitlab.com's API Puma instances

When this file is uploaded as a Junit artifact:

Ci::BuildFinishedWorker tries to read the file to parse the test report and then runs out of memory and crashes
The artifact file is still stored!
Unauthenticated requests (in public projects) can be made to the test_reports API endpoint, and that causes Puma to try to unzip and read the same artifact file, which then again leads to high CPU usage and OOMKills.

What is the expected correct behavior?

The amount of data read from GzipStream should be limited. Even though report files can be large, it makes no sense to allow multi-gigabyte files to be parsed.

Relevant logs and/or screenshots

See the attached video, it includes some logs.

Output of checks

This bug has not been tested with Gitlab.com. I assume it may still have some effect there. At least the Sidekiq crashes should be reproducible.

A full Puma DoS might be more difficult to achieve due to the number of Puma instances running. But then again, as the attack can be triggered by unauthenticated requests, an attacker could just setup a couple of dozen machines doing the API requests and maybe that could be enough?

Results of GitLab environment info

Docker installation:

###  gitlab-rake gitlab:env:info    
  

System information    
System:    
Proxy:          no    
Current User:   git    
Using RVM:      no    
Ruby Version:   2.7.5p203    
Gem Version:    3.1.6    
Bundler Version:2.3.15    
Rake Version:   13.0.6    
Redis Version:  6.2.7    
Sidekiq Version:6.4.2    
Go Version:     unknown    
    
GitLab information    
Version:        15.5.4-ee    
Revision:       d3dda7548e0    
Directory:      /opt/gitlab/embedded/service/gitlab-rails    
DB Adapter:     PostgreSQL    
DB Version:     13.6    
URL:            http://gl.lkoskela.com:8929    
HTTP Clone URL: http://gl.lkoskela.com:8929/some-group/some-project.git    
SSH Clone URL:  ssh://git@gl.lkoskela.com:2224/some-group/some-project.git    
Elasticsearch:  no    
Geo:            no    
Using LDAP:     no    
Using Omniauth: yes    
Omniauth Providers:      
    
GitLab Shell    
Version:        14.12.0    
Repository storage paths:    
- default:      /var/opt/gitlab/git-data/repositories    
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Impact

API / Puma DoS

After uploading a malicious test report artifact file, an attacker can cause Gitlab's web service to try to parse that file with unauthenticated requests. This causes to web service to use excessive CPU resources and then crash due to running out of memory. These requests can be repeated to sustain the attack. Gitlab's API becomes unresponsive and service is denied for all users

The severity of this depends on the web server (Puma) setup: with larger and more distributed instances it will be more difficult to make all instances crash. In small self-hosted environments though this can have a large impact on the functionality of Gitlab. Plus, as the attack requires no authentication, there's DDoS potential here.

Sidekiq DoS

An attacker can get a sidekiq worker OOMKilled by a simple file upload. This will interrupt any background jobs running on that particular worker. Because the attack is very simple, the attacker can do this often to continuously cause crashes.

The severity of this depends on the sidekiq setup: with larger and more distributed instances it will of course be smaller as crashes are limited to only a subset of sidekiq instances. In small self-hosted environments though this can have a large impact on the functionality of Gitlab.

This can affect any user in the Gitlab instance because much of Gitlab's functionality relies on sidekiq jobs. Background job execution may get delayed or in some cases they may not get executed at all (if attacker can keep Sidekiq crashing continuously). For instance, this may cause a CI pipeline to fail and be left in a "pending" state for a long time, if a background job for that pipeline was running when sidekiq crashed.

Attachments

Warning: Attachments received through HackerOne, please exercise caution!

How To Reproduce

Please add reproducibility information to this section: