Skip to content

Puma DOS via XML manifest file import (/import/manifest/new)

⚠️ Please read the process on how to fix security issues before starting to work on the issue. Vulnerabilities must be fixed in a security mirror.

HackerOne report #2571364 by a92847865 on 2024-06-24, assigned to @greg:

Report | Attachments | How To Reproduce

Report

NOTE! Thanks for submitting a report! Please note that initial triage is handled by HackerOne staff. They are identified with a HackerOne triage badge and will escalate to the GitLab team any. Please replace all the (parenthesized) sections below with the pertinent details. Remember, the more detail you provide, the easier it is for us to triage and respond quickly, so be sure to take your time filling out the report!

Summary

GitLab allows you to import all the required Git repositories based on a manifest xml file. Because the import manifest file is parsed by Nokogiri::DOM parser directly without any size check, an attacker can upload a big file to cause memory exhaustion on Gitlab instance.

This vulnerability affects instances with Gitlab Cloud Native Hybrid (similar to Gitlab.com settings) and normal settings also:

  • Gitlab Cloud Native Hybrid: Each Webservice pod (Puma and Workhorse) is recommended to be run with the following configuration : 4 Puma Workers, 4 vCPU, 5 GB memory (request), 7 GB memory (limit). An attacker needs to send only one request to cause one Webservice pod to exceed memory limit. For medium size instances, which serve 200 RPS or 10,000 users and have around 20 Webservice pods, only 20 requests are needed to force all pods to exceed memory limit. It usually takes seconds to dozens of seconds to restart those pods --> service outage. Attacker can continue to send import requests to maintain service outage.
  • Normal (non Cloud-Native): For medium size instances, which serve 200 RPS or 10,000 users and have 3 Rails nodes (each node needs 32 vCPU, 28.8 GB memory), attacker has two options: increasing payload size accordingly to send less requests or keeping the same payload size and sending more request. The DOS impact on non Cloud-Native instances is more severe than Cloud-Native instances because by default there is no mechanism to automatically restart a node. A malfunctioned node is likely needed to restart manually --> longer duration of service outage. --> less requests are needed to maintain service outage.
Steps to reproduce
  1. Register a VM (8GB RAM).
  2. Install Gitlab EE.
  3. Sign in to your Gitlab instance and enable manifest file import (enable by default on Gitlab.com).
  4. Create a group which will be used in step 6.
  5. Create payload (import manifest file). You can use pwn.xml or create one using payload_generation.py.
  6. Browse to http://$your_GL_instance/import/manifest/new. Choose group you've created in step 4 as target import group, upload payload you've created in step 5 as import manifest file and click "List available repositories". You can find detail instructions here.
  7. SSH to your VM, run htop command to monitor memory usage of your VM. After around 1 minute, your VM will exceed its memory limit.
  8. Browse to your Gitlab instance, try to click some buttons, you'll notice that your Gitlab instance is totally unresponsive.
Impact

An attacker can send small number of manifest import requests to cause Puma web service outage to other Gitlab users.

Examples

(If the bug is project related, please create an example project and export it using the project export feature)

(If you are using an older version of GitLab, this will also help determine whether the bug has been fixed in a more recent version)

(If the bug can be reproduced on GitLab.com without violating the Rules of Engagement as outlined in the program policy, please provide the full path to the project.)

What is the current bug behavior?

Manifest file is passed to Gitlab::ManifestImport::Manifest.new here:
manifest = Gitlab::ManifestImport::Manifest.new(params[:manifest].tempfile)
And then being parsed by Nokogiri::DOM parser directly without any size checkhere:

def initialize(file)  
        [@]parsed_xml = Nokogiri::XML(file) { |config| config.strict }  
        [@]errors = []  
      rescue Nokogiri::XML::SyntaxError  
        [@]errors = ['The uploaded file is not a valid XML file.']  
      end
What is the expected correct behavior?

The import manifest file's size should be checked before being parsed by Nokogiri::DOM parser.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output,
logs, and code as it's very hard to read otherwise.)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info
System information  
System:         Ubuntu 20.04  
Proxy:          no  
Current User:   git  
Using RVM:      no  
Ruby Version:   3.1.5p253  
Gem Version:    3.5.11  
Bundler Version:2.5.11  
Rake Version:   13.0.6  
Redis Version:  7.0.15  
Sidekiq Version:7.1.6  
Go Version:     unknown

GitLab information  
Version:        17.1.0-ee  
Revision:       b7514f9c21c  
Directory:      /opt/gitlab/embedded/service/gitlab-rails  
DB Adapter:     PostgreSQL  
DB Version:     14.11  
URL:            http://34.16.168.95  
HTTP Clone URL: http://34.16.168.95/some-group/some-project.git  
SSH Clone URL:  git@34.16.168.95:some-group/some-project.git  
Elasticsearch:  no  
Geo:            no  
Using LDAP:     no  
Using Omniauth: yes  
Omniauth Providers: 

GitLab Shell  
Version:        14.36.0  
Repository storages:  
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket  
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly  
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket  
- default Version:      17.1.0  
- default Git Version:  2.45.1

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)

(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Impact

An attacker can send a relatively small number of manifest import requests to cause total Puma web service outage on Gitlab self-hosted instances and potentially Gitlab.com which is a Gitlab Cloud Native Hybrid instance.

Attachments

Warning: Attachments received through HackerOne, please exercise caution!

How To Reproduce

Please add reproducibility information to this section:

Edited by Félix Veillette-Potvin