Skip to content

Denial-of-Service via Bulk Issue References in User-Controllable Fields

Note_ regarding who owns this Issue:_

The bulk of the report seems to indicate the act of trying to close a bazillion issues and overloading sidekiq workers, which has nothing to do with the markdown. So if it's the reference detection, it would be ~group:knowledge, and if it's the closing behavior, that would be groupproject management.

Chances are it will require attention in both areas.

⚠️ Please read the process on how to fix security issues before starting to work on the issue. Vulnerabilities must be fixed in a security mirror.

HackerOne report #2903896 by pwnie on 2024-12-17, assigned to @ameyadarshan:

Report | How To Reproduce

Report

Summary

GitLab’s processing of GFM references in commit messages, merge request (MR) descriptions, and notes allows attackers to reference an arbitrarily large number of issues, merge requests, or commits. When these references are processed, GitLab attempts to resolve and update all mentioned objects simultaneously. By including thousands of references, an attacker can force Sidekiq workers to consume excessive memory, eventually triggering the Linux Out-of-Memory (OOM) killer and causing a denial-of-service (DoS) across the entire GitLab instance’s background job processing.

This vulnerability stems from the logic in Mentionable and related code (e.g., ProcessCommitWorker, Gitlab::ReferenceExtractor, Notes::PostProcessService), which parses references without imposing practical limits. When triggered by a single push, MR update, or note creation, this can halt background tasks, CI pipelines, and other essential operations.

Details

Affected Components:

  • Mentionable module (Mentionable#all_references): Extracts references from fields like issue descriptions, MR descriptions, commit messages, and notes.
  • Gitlab::ReferenceExtractor: Resolves every mentioned reference, whether it's an issue, MR, or commit.
  • ProcessCommitWorker and Notes::PostProcessService: Invoke create_cross_references!, which ultimately calls referenced_mentionables and extracted_mentionables. Any user-supplied text processed here can contain massive references.
  • Sidekiq Workers: Spawn multiple jobs to close or process all referenced issues. Thousands of references result in excessive memory consumption.

Code Flow Example:

  1. User Input (Commit Message or Note):
    A user pushes a commit or posts a note with content referencing a large number of issues, e.g.:

    Close #1 #2 #3 ... #2000

  2. Reference Extraction:
    When GitLab processes this input (e.g., via ProcessCommitWorker or when saving a note and running Notes::PostProcessService), it calls create_cross_references!, which uses Mentionable#all_references to parse all mentioned issues, MRs, and commits.

  3. Unbounded Parsing:
    Gitlab::ReferenceExtractor attempts to resolve thousands of references at once. This leads to large data structures in memory, database lookups, and the creation of numerous Sidekiq jobs.

  4. Memory Exhaustion:
    The Sidekiq process consumes massive memory to handle all these references. With enough references (e.g., thousands of large issues), the OOM killer terminates the Sidekiq process.

  5. DoS Condition:
    Without Sidekiq, critical background tasks (CI, notifications, housekeeping) stall, causing a severe availability impact.

Note on Reaching extracted_mentionables:
Even a single note on an issue can trigger this logic. When a user creates or updates a note referencing numerous issues, Notes::PostProcessService runs after saving that note and calls note.create_cross_references!. This leads to referenced_mentionables → extracted_mentionables and ultimately tries to load and process every mentioned object, amplifying the memory usage and leading to DoS.

Steps to Reproduce

  1. Set Up a Large Number of Issues:

    • On a GitLab instance (self-hosted or gitlab.com scale), create thousands of issues. Ensure the environment has limited memory headroom (e.g., 8GB total memory with ~2GB available to GitLab processes).

  2. Craft a Malicious Commit or Note:

    • Via the GitLab Web IDE or the "Edit File" interface, modify a file and set the commit message to reference all issues:

      Close #1 #2 #3 ... #2000

    • Commit these changes directly to the default branch or a merge request.

    (Alternatively, via the command line):

    git commit -m "Close #1 #2 #3 ... #2000" --author="Attacker <attacker@example.com>"  
git push

    Or add a note to an existing issue with a large number of references in its comment box on the issue page (e.g., “Check these: #1 (closed) #2 (closed) #3 (closed) ... #2000 (closed)”).

  3. Trigger Processing:

    • Once pushed or saved, GitLab’s background processes attempt to close all referenced issues at once.
    • This leads to a massive spike in Sidekiq’s memory usage.

  4. Observe the Result:

    • Check cat /var/log/gitlab/sidekiq/current | grep -i terminate for evidence of Sidekiq worker termination.
    • Confirm that CI and other background tasks are no longer running due to the OOM-induced DoS.
Impact
  • Availability Loss:
    The OOM termination of Sidekiq suspends all background job processing, affecting CI pipelines, notifications, repository housekeeping, and more.
  • Widespread Disruption:
    Users experience delays, stalled pipelines, and a system unable to handle routine maintenance tasks, effectively partial downtime.
  • Recovery Overhead:
    Administrators need to restart workers, possibly scale infrastructure, and deal with manually reopening issues or restoring normal operations, which may be challenging if thousands of issues were closed en masse.
Suggested Remediation
  • Input Validation & Limit Enforcement:
    Impose strict upper bounds on the number of references processed per commit, merge request, or note. If references exceed a certain threshold, ignore or truncate the list.
  • Batch Processing:
    Instead of processing all references in a single go, batch them into manageable chunks to keep memory consumption in check.
  • Early Checks:
    Before creating Sidekiq jobs, validate that the number of references is within safe limits and fail or partially process if it’s excessive.
Related Issues

This vulnerability is similar in nature to previously reported resource exhaustion issues (like Report #2401952). By adjusting the CVSS ratings and addressing the underlying cause (unbounded reference handling), GitLab ensures both IDOR and DoS aspects are covered by the same set of mitigations.

By addressing these concerns, GitLab can prevent a single user’s malicious commit message, MR description, or note from causing widespread denial-of-service and ensure stability and reliability for all users.

Impact

Impact
Availability Loss:

The OOM termination of Sidekiq suspends all background job processing, affecting CI pipelines, notifications, repository housekeeping, and more.
Widespread Disruption:

Users experience delays, stalled pipelines, and a system unable to handle routine maintenance tasks, effectively partial downtime.
Recovery Overhead:

Administrators need to restart workers, possibly scale infrastructure, and deal with manually reopening issues or restoring normal operations, which may be challenging if thousands of issues were closed en masse.

How To Reproduce

Please add reproducibility information to this section:

Edited by Matthew Macfarlane