Skip to content

Enables parsing nested sections in logs

What does this MR do and why?

This change allows the logs page to display nested log sections.

This is possible when the GitLab Runner produces nested log sections and subsections, such as when multiline commands can be collapsed.

Changelog: changed

How does this work?

Previously, our logs were grouped into sections before rendering them in the frontend. In this design each section had a isClosed flag that determined if the logs in a given section should be shown. This made having nested sections difficult: We would have to render separate sections in their own data structure and we would have recurse through them to render.

This change separates the "sections" information from the "log lines", so the sections are presented alongside the log lines. A section's isClosed state still determines if a section is shown, but each line gets rendered independently in the HTML and shown/hidden according to the state of the section it belongs to.

Why this change?

This change implements #409350 (closed), which is needed to render job logs correctly when a runner has FF_SCRIPT_SECTIONS, which can be nested inside other sections.

Preparation refactor Adds explicit section footers to logs and harde... (!136170 - merged)
Log format change Enables parsing nested sections in logs (!136186 - merged) You are here

Screenshots or screen recordings

Before After
2023-11-09_12.01.24 2023-11-09_11.57.08

I also tested on other use cases, to ensure we don't introduce any bugs:

job that gets cancelled job with a section partially shown a vanilla command
2023-11-09_11.58.26 2023-11-09_11.59.30 2023-11-09_12.00.39

How to set up and validate locally

  1. Ensure you have installed GitLab Runner v16.5.0+.
  2. In a project, create the following pipeline
variables:
  FF_SCRIPT_SECTIONS: "true" # this will allow you to collapse arbitrary scripts in the log

stages:
  - build

simple-job:
  stage: build
  script:
    - echo 'A simple command'

# A multi-line command that should be collapsible
multi-line-command-job:
  stage: build
  script:
    - |
      echo "{
        'test': 'data',
        'test2': 'data2',
      }"
    - |
      echo "{
        'test': 'data',
        'test2': 'data2',
      }"

# This jobs helps me test long running jobs with many lines that gets cancelled
# You'll have to cancel this job manually to truncate it.
long-running-job:
  stage: build
  script:
    - while sleep 1; do date; done

# A job that triggers "Showing last 500.04 KiB of log" at the top for being too long
showing-last-job:
  stage: build
  script:
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done
    - for i in {1..500}; do echo "This is a long line that gets repeated many many many many many many many many many many many many times $i"; done

collapsible_job_single:
  variables:
    FF_SCRIPT_SECTIONS: "false"
  stage: build
  script:
    - echo -e "\033[0Ksection_start:`date +%s`:my_first_section\r\033[0KHeader of the 1st collapsible section"
    - echo 'this line should be hidden when collapsed'
    - echo -e "\033[0Ksection_end:`date +%s`:my_first_section\r\033[0K"

collapsible_job_multiple:
  variables:
    FF_SCRIPT_SECTIONS: "false"
  stage: build
  script:
    - echo -e "\033[0Ksection_start:`date +%s`:my_first_section\r\033[0KHeader of the 1st collapsible section"
    - echo 'this line should be hidden when collapsed'
    - echo -e "\033[0Ksection_start:`date +%s`:second_section\r\033[0KHeader of the 2nd collapsible section"
    - echo 'this line should be hidden when collapsed'
    - echo -e "\033[0Ksection_end:`date +%s`:second_section\r\033[0K"
    - echo -e "\033[0Ksection_end:`date +%s`:my_first_section\r\033[0K"
  1. Check the job output of each and confirm they are correct

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #409350 (closed)

Edited by Miguel Rincon

Merge request reports