Scheduled pipeline execution policy job fails to download artifacts from a previous job

Summary

Reported by @kumaresh.s in Slack (internal):

With scheduled pipeline execution policies, a dependency_scanning job in a test stage fails to start due to missing permissions to download artifacts of a previous job from .pipeline-policy-pre stage.

Steps to reproduce

Example Project

PEP YAML:

# This configuration generates Gradle lock files
# and runs dependency scanning

stages:
  - .pipeline-policy-pre  # Special stage that runs BEFORE project stages
  - build
  - test

# Job to generate Gradle lock files
gradle_lockfile_generation:
  stage: .pipeline-policy-pre
  image: gradle:9.1-jdk25
  
  # Only run this job for projects with Gradle files
  rules:
    - if: '$CI_COMMIT_BRANCH'
      exists:
        - '**/build.gradle'
        - '**/build.gradle.kts'
  
  before_script:
    # Create the Nebula plugin initialization script
    - |
      cat << 'EOF' > nebula.gradle
      initscript {
        repositories { 
          mavenCentral() 
        }
        dependencies {
          classpath 'com.netflix.nebula:gradle-dependency-lock-plugin:16.0.1'
        }
      }
      allprojects {
        apply plugin: nebula.plugin.dependencylock.DependencyLockPlugin
      }
      EOF
    # Make gradlew executable if it exists
    - chmod +x gradlew 2>/dev/null || echo "No gradlew found, will use system gradle"
  
  script:
    # Generate lock file with transitive dependencies
    - |
      if [ -f "./gradlew" ]; then
        ./gradlew --init-script nebula.gradle \
          -PdependencyLock.includeTransitives=true \
          -PdependencyLock.lockFile=dependencies.lock \
          generateLock saveLock
      else
        gradle --init-script nebula.gradle \
          -PdependencyLock.includeTransitives=true \
          -PdependencyLock.lockFile=dependencies.lock \
          generateLock saveLock
      fi
    
    # Generate lock file with direct dependencies only
    - |
      if [ -f "./gradlew" ]; then
        ./gradlew --init-script nebula.gradle \
          -PdependencyLock.includeTransitives=false \
          -PdependencyLock.lockFile=dependencies.direct.lock \
          generateLock saveLock
      else
        gradle --init-script nebula.gradle \
          -PdependencyLock.includeTransitives=false \
          -PdependencyLock.lockFile=dependencies.direct.lock \
          generateLock saveLock
      fi
    
    # Verify lock files were created
    - find . -name "dependencies*.lock" -type f
  
  after_script:
    # Clean up temporary build artifacts to save space
    - find . -path '*/build/dependencies*.lock' -delete
  
  # Export lock files as artifacts for dependency scanning job
  artifacts:
    paths:
      - '**/dependencies.lock'
      - '**/dependencies.direct.lock'
    expire_in: 1 day
  
  allow_failure: false

# Include dependency scanning
include:
  - component: $CI_SERVER_FQDN/components/dependency-scanning/main@0

# Enable the new SBOM-based analyzer
variables:
  DS_ENFORCE_NEW_ANALYZER: "true"

What is the current bug behavior?

The job fails to start due to 403 Forbidden - This action cannot be performed by internal users which happens when trying to download artifacts of a previous job gradle_lockfile_generation.

The job works fine when it's executed as a regular user.

What is the expected correct behavior?

The permissions should be granted for the policy bot user to download the artifacts.

Relevant logs and/or screenshots

https://gitlab.com/gl-demo-ultimate-kumaresh/demos/java/simple-java-gradle-app/-/jobs/11785690029

Job logs
Running with gitlab-runner 18.4.0~pre.195.g5de42c65 (5de42c65)
  on blue-1.saas-linux-small.runners-manager.gitlab.com/default j1aLDqxSn, system ID: s_ccdc2f364be8
Resolving secrets
Preparing the "docker+machine" executor 00:09
Using Docker executor with image registry.gitlab.com/security-products/dependency-scanning:v0 ...
Using effective pull policy of [always] for container registry.gitlab.com/security-products/dependency-scanning:v0
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/security-products/dependency-scanning:v0 ...
Using docker image sha256:599c80b67eb7f0ae923949bcc838551455ff76c13112ed6e1c179705b813cf38 for registry.gitlab.com/security-products/dependency-scanning@sha256:0d4972b6dfbc76796adecdd1f29004ff67f51c07675a1c7ce53e2dc89b36bee3 ...
Preparing environment 00:04
Using effective pull policy of [always] for container sha256:1c7ef7eba67dc4ec18094f5f5c67fcdc7bb9c4a15e08ca51ca3b6177e2e43f0c
Running on runner-j1aldqxsn-project-75092203-concurrent-0 via runner-j1aldqxsn-s-l-s-amd64-1761033359-4e4a3a80...
Getting source from Git repository 00:01
Gitaly correlation ID: e00e64deb3324c56b0d37713b27e6b4d
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/gl-demo-ultimate-kumaresh/demos/java/simple-java-gradle-app/.git/
Created fresh repository.
Checking out 49678023 as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}" || echo 'Not a git repository; skipping'
Downloading artifacts 00:01
Downloading artifacts for gradle_lockfile_generation (11785690009)...
ERROR: Downloading artifacts from coordinator... forbidden  correlation_id=208189ca0d074597819c8e8a55f5055f host=us-east1-c.ci-gateway.int.gprd.gitlab.net:8989 id=11785690009 responseStatus=403 Forbidden status=GET https://us-east1-c.ci-gateway.int.gprd.gitlab.net:8989/api/v4/jobs/11785690009/artifacts: 403 Forbidden (403 Forbidden - This action cannot be performed by internal users) token=eyJraWQiO
FATAL: permission denied                           
Retrying in 5s
Uploading artifacts for failed job 00:00
Uploading artifacts...
WARNING: **/gl-sbom-*.cdx.json: no matching files. Ensure that the artifact path is relative to the working directory (/builds/gl-demo-ultimate-kumaresh/demos/java/simple-java-gradle-app) 
ERROR: No files to upload                          
Uploading artifacts...
WARNING: gl-dependency-scanning-report.json: no matching files. Ensure that the artifact path is relative to the working directory (/builds/gl-demo-ultimate-kumaresh/demos/java/simple-java-gradle-app) 
ERROR: No files to upload                          
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

(For installations from source run and paste the output of: sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)

(we will only investigate if the tests are passing)

Implementation Plan

To resolve this issue, the security policy bot permissions need to be refactored and enhanced through the following steps:

Phase 1: Refactor to YAML Format

Task: #594740 (closed) - Refactor security policy bot permissions to use YAML format

Migrate all security policy bot authorization policies from Ruby-based policy classes to the new YAML format. This establishes a foundation for managing bot permissions in a more maintainable and transparent way.

Deliverables:

  • Create config/authz/roles/security_policy_bot.yml with all current permissions
  • Migrate existing Ruby policy classes to YAML format
  • Ensure backward compatibility during transition

Phase 2: Update to Minimal Access

Task: #594741 (closed) - Update security policy bot role to minimal access

Replace the guest role with a minimal access role that only includes permissions necessary for the bot to function. This will be implemented behind a feature flag to allow testing and validation.

Deliverables:

  • Implement feature flag security_policy_bot_minimal_access
  • Update bot creation logic to use minimal access when flag is enabled
  • Identify and document required permissions through testing
  • Maintain backward compatibility with guest role as fallback

Phase 3: Grant API Access

Task: #594742 - Grant API access to security policy bot role

Once the bot has been migrated to minimal access, grant the necessary API permissions to allow the bot to download artifacts and perform other required operations.

Deliverables:

  • Add API access permissions to the security_policy_bot role
  • Document which API endpoints the bot needs access to
  • Test API access in the minimal access context
  • Verify no additional permissions are needed

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖