Skip to content

Backend: Unbounded check in build trace section timestamp prevents a CI trace from being loaded

HackerOne report #1587261 by exem_pt on 2022-05-31, assigned to @cmaxim:

Report | How To Reproduce

Report

NOTE! Thanks for submitting a report! Please replace all the (parenthesized) sections below with the pertinent details. Remember, the more detail you provide, the easier it is for us to triage and respond quickly, so be sure to take your time filling out the report!

Summary

If an attacker has control over the printed contents of a CI job trace, either via access to the .gitlab-ci.yml file, or as part of a related supply chain security issue, they may prevent access to the CI job trace, effectively hiding their tracks. They may do this by using the "custom collapsible sections" feature: https://docs.gitlab.com/ee/ci/jobs/#custom-collapsible-sections with an out-of-bounds timestamp.

Steps to reproduce
  1. Create a .gitlab-ci.yml file with the following contents:
stages:  
  - build

build-job:  
  stage: build  
  before_script: |  
    timed() {  
      timedcnt=$((timedcnt+1))  
      echo -e "section_start:$(date +%s):timed_$timedcnt\r\e[0KTimed command $timedcnt"  
      "$@"  
      echo -e "section_end:9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999:timed_$timedcnt\r\e[0K"  
    }  
    sleeptalk() {  
      sleep $1  
      shift  
      echo "$@"  
    }  
  script: |  
    timed sleeptalk 5 Five bottles of beer on the wall, five bottles of beer, take one down and pass it around, four bottles of beer on the wall  
    timed sleeptalk 4 Four bottles of beer on the wall, four bottles of beer, take one down and pass it around, three bottles of beer on the wall  
    timed sleeptalk 3 Three bottles of beer on the wall, three bottles of beer, take one down and pass it around, two bottles of beer on the wall  
    timed sleeptalk 2 Two bottles of beer on the wall, two bottles of beer, take one down and pass it around, one bottle of beer on the wall  
    timed sleeptalk 1 One bottle of beer on the wall, one bottle of beer, take one down and pass it around, no more bottles of beer on the wall  
    timed sleeptalk 0 No more bottles of beer on the wall, no more bottles of beer, go to the store and buy some more, five bottles of beer on the wall
  1. Wait for the job to finish
  2. Attempt to load the job output
  3. See that the trace.json network request (such as https://gitlab.com/xxx/yyy/-/jobs/zzz/trace.json?state=) will return a 500, preventing the job trace from being displayed
Impact

It's possible for Developer-role and above users with access to the .gitlab-ci.yml file to prevent their job trace from being reviewed/audited by other people, allowing them to perform actions inside a job that may otherwise go unscrutinised.

Additionally, if there was a supply-chain issue, if unmitigated, this could be a common way for attackers that compromise the supply chain to go undetected for longer (as an example, if someone took over an npm package and made it echo out the relevant section_start/section_end parts and that content was in a GitLab CI/CD job, you would not be able to see it due to the HTTP 500 error).

Examples

An example job was run here: https://gitlab.com/davebarr/ci-testing/-/jobs/2526993347 - this project is currently set to Private. If you need access, let me know, but the example .gitlab-ci.yml file above should be make this easy to replicate.

What is the current bug behavior?

https://gitlab.com/davebarr/ci-testing/-/jobs/2526993347 fails to load.

What is the expected correct behavior?

https://gitlab.com/davebarr/ci-testing/-/jobs/2526993347 loads successfully.

Output of checks

This bug happens on GitLab.com

Impact

It's possible for Developer-role and above users with access to the .gitlab-ci.yml file to prevent their job trace from being reviewed/audited by other people, allowing them to perform actions inside a job that may otherwise go unscrutinised.

Additionally, if there was a supply-chain issue, if unmitigated, this could be a common way for attackers that compromise the supply chain to go undetected for longer (as an example, if someone took over an npm package and made it echo out the relevant section_start/section_end parts and that content was in a GitLab CI/CD job, you would not be able to see it due to the HTTP 500 error).

How To Reproduce

Please add reproducibility information to this section:

Possible solution

Limit the duration value before extracting the time parts:

diff --git a/lib/gitlab/ci/ansi2json/line.rb b/lib/gitlab/ci/ansi2json/line.rb
index e48080993ab..25c38ac5247 100644
--- a/lib/gitlab/ci/ansi2json/line.rb
+++ b/lib/gitlab/ci/ansi2json/line.rb
@@ -80,7 +80,7 @@ def set_as_section_header
         end
 
         def set_section_duration(duration_in_seconds)
-          duration = ActiveSupport::Duration.build(duration_in_seconds.to_i)
+          duration = ActiveSupport::Duration.build(duration_in_seconds.to_i.clamp(0, 1.year))
           hours = duration.in_hours.floor
           hours = hours > 0 ? "%02d" % hours : nil
           minutes = "%02d" % duration.parts[:minutes].to_i

I think an upper limit of 1 year is more than sufficient since the jobs are timing out after a few hours.

Edited by Marius Bobin