Skip to content

Log the included file count of CI pipelines that exceed the limit

Leaminn Ma requested to merge add-pipeline-includes-logging into master

What does this MR do and why?

In #367150 (closed), we discovered a bug that allows the includes limit (currently 100) to be bypassed. In order to gauge how many customers would be affected by enforcing the limit, this MR logs the included file count of pipelines that currently exceed the limit.

In this MR, we leverage the existing pipeline creation logger infrastructure. The logs are outputted to gitlab/log/application_json.log.

How to set up and validate locally

  1. Enable the logging feature flag locally:
bundle exec rails c
Feature.enable(:ci_pipeline_creation_logger)
  1. So that we don't have to trigger the log by exceeding 100 files, update the limit to 1 in lib/gitlab/ci/config/external/context.rb.
MAX_INCLUDES = 1
  1. Tail the log file in your gitlab directory:
tail -f log/application_json.log
  1. Create a template1.yml file in your project root with the following content:
job:
    script: echo "test"
  1. Update your .gitlab-ci.yml with the following content:
include:
    - local: 'template1.yml'
  1. Run the pipeline and observe that a log containing "pipeline_includes_count":2 is produced. Note that the count should be 2 because it counts itself (.gitlab-ci.yml) as well as in the included local file (template1.yml).

Screenshot_2023-01-10_at_9.25.15_AM

  1. (Optional). You can also observe that if you update .gitlab-ci.yml without an include file, a log entry is not produced as the file count does not exceed the limit (1).

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #367150 (closed)

Edited by Leaminn Ma

Merge request reports