Skip to content

Extend "needs:artifacts" syntax to download dotenv artifacts only

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem to solve

TL;DR:

Our pipelines spend a huge amount of time downloading artifacts they don't actually need, because we cannot specify which type of artifacts to download.

Long version:

In our pipeline we have a build job which produces large build artifacts (~ 2 GB). A later job in the pipeline depends on it, because it needs some variables from it. This is a minimal example (using an 800 MB artifact, because there is a size limit of 1 GB on gitlab.com):

build:
  stage: build
  script:
    - truncate -s 800M artifact.bin
    - echo "VARIABLE=value" > build.env
  artifacts:
    reports:
      dotenv: build.env
    paths:
      - artifact.bin

use_variable:
  stage: use_variable
  variables:
    - COMPLEX_VARIABLE="complex-$VARIABLE"
  script:
    - echo "$COMPLEX_VARIABLE" # displays "complex-value"
  needs:
    - build

The use_variable job however does not need the build artifacts. We have the possibility to skip artifact downloading like this:

  needs:
    - job: build
      artifacts: false

However, this also prevents the build.env from being downloaded, resulting in $VARIABLE not being available. There is no way to have the dotenv information passed to a subsequent job without downloading the complete artifacts. Downloading gigabytes of unnecessary data slows down pipeline speed significantly, though.

Proposal

Extend the needs:artifacts syntax so that we can choose which artifacts to download. This could probably look like this:

  needs:
    - job: build
      artifacts:
        - archive
        - dotenv

Specifying archive only would download the actual artifacts only, specifying dotenv only would download the build.env file only. The new syntax could exist in parallel to the old syntax, so that this won't affect existing pipelines.

Since there are lots of more artifacts reports types, those might be added to the list of possible values as well if that makes sense (I actually don't use any of those and have no idea if they might have any useful value in subsequent jobs).

Edited by 🤖 GitLab Bot 🤖