Skip to content

Restrict access to jobs logs when CI_DEBUG_TRACE=true is set via Runner

In GitLab 13.7 and later, we restricted the ability to see job logs to users with developer or higher permissions when debug logging is enabled(CI_DEBUG_TRACE=true) in YAML defined or GitLab UI defined variables.

However, users without developer or higher permissions are still able to access job logs if the variable is set to True directly in Runner or anywhere outside of GitLab and the ci config files.

On the original issue @grzesiek said:

Probably the most simple fix here is to check if CI_DEBUG_TRACE is set on the GitLab side, this might be the quick fix, but in order to fix that properly I think that the runner should send this information the it is a detailed trace.

@tmaczukin said:

In next step we can create another job metadata field like trace_debug boolean (not sure where to put it, since ci_builds is already too big...) that would be set to true on detection of CI_DEBUG_TRACE and the permission would act depending on this field.

And as the third step, we could extend the Runner API for the trace update or trace patch, to send information about trace_debug=(true|false) depending on what's detected on the Runner side (since at the end the decision about enabling debug trace is happening in Runner). This could then cover all possible cases where CI_DEBUG_TRACE can be set

Proposal

Hide debug logs from developers or above even if CI_DEBUG_TRACE=true is set only on the runner side by passing back that information and storing it alongside the metadata.

Ensure that the security vulnerability described here is addressed: #375180 (closed)

Technical

Change the debug_mode function to check a new persisted column stored for each build that is populated via the runner API from the state of the variables in runner.

In this issue we will implement changing the permission checks to look at the new build related column rather than the variables set in gitlab rails. We can do this via the security developer process.

We must ensure that when the CI_DEBUG_TRACE is set to "1", "t", "T", "true", "TRUE", "True" the logs stay hidden. More details around that vulnerability here: #375180 (closed). We can fall back on the parsing the variables from gitlab if data is not received from the runner(they are not on the correct version) but change debug_mode? to consider "1", "t", "T", "true", "TRUE", "True"astrue`.

Implementation notes

Trace patch endpoint changes

We need to make a change to how debug_trace is saved when it is sent to the append trace endpoint.

Runner will send the debug_trace value as a boolean on the first append or whenever <code data-sourcepos="36:89-36:102">CI_DEBUG_TRACE</code> or <code data-sourcepos="36:109-36:125">CI_DEBUG_SERVICES</code> changes to <code data-sourcepos="36:140-36:143">true</code>, and currently we only save on first append . We should update the append trace endpoint to also save if debug_trace is not null.

Build model changes

Then in app/models/ci/build.rb we can change the debug_mode? check to look at the value on both the gitlab and runner side. This will ensure backwards compatibility with older versions of runner.

    def debug_mode?
      # perform the check on both sides in case the runner version is old
      [
        "1", "t", "T", "true", "TRUE", "True"
      ].include?(variables['CI_DEBUG_TRACE']&.value) || trace_metadata.debug_trace
    end

We check debug_mode before allowing access to the logs.

Project Application controller changes

We currently have authorize_read_build_trace! in the Projects::ApplicationController where we set a custom error message that needs to be updated to reflect that the variables value is respected on the runner side and not just what is set in the yaml file or on the gitlab side. I've also made a small tech debt issue to move this code #387676 (closed).

Documentation

We need to update the documentation here: https://docs.gitlab.com/ee/ci/variables/#restrict-access-to-debug-logging

Edited by Allison Browne