Skip to content
Snippets Groups Projects

Job dependency fail with error about expired artifacts when artifacts are expired and locked

    • Closed Issue created by Kristian Berge Nessa @cavatina

    Summary

    When Job B has a needs: dependency on Job A, retrying Job B fails with error:

    This job depends on other jobs with expired/erased artifacts: Job A Please refer to https://docs.gitlab.com/ee/ci/yaml/README.html#dependencies

    if the artifacts from job B was supposed to be expired by expire_in rules, but were not due to "latest" policy;

    Job A show this message, and its artifacts are in fact available:

    These artifacts are the latest. They will not be deleted (even if expired) until newer artifacts are available.

    What's puzzling is that even if job A is retried, and successfully run so that new artifacts become available, retrying job B still yields the same error.

    Steps to reproduce

    1. Define Job A with artifacts that has a short expire_in.
    2. Define Job B in another stage, that has a needs dependency on Job A. Let there be an error so the job will fail.
    3. Run the pipeline.
    4. Let enough time pass that the artifacts from Job A would normally expire.
    5. Verify that the artifacts from Job A are in fact still available due to being latest for the job/branch.
    6. Retry Job B.
    7. Job B will fail with the above error (This job depends on other jobs with expired/erased artifacts).

    Example Project

    Happens on self hosted GitLab instance (13.6.3).
    The error has also been present on earlier versions.

    build:
  stage: build
  script:
    - echo "this is a build job" >> build.txt
  artifacts:
    paths:
      - build.txt
    expire_in: 10 seconds

deploy:
  stage: deploy
  script:
    - cat build.txt
  when: manual

    What is the current bug behavior?

    The Job with a needs dependency fail with error:

    This job depends on other jobs with expired/erased artifacts: Job A Please refer to https://docs.gitlab.com/ee/ci/yaml/README.html#dependencies

    What is the expected correct behavior?

    The Job with a needs dependency would run, when the artifacts are in fact available.

    Relevant logs and/or screenshots

    Output of checks

    Results of GitLab environment info

    Expand for output related to GitLab environment info 
    System information
System:         Debian 10
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.1
Redis Version:  5.0.9
Git Version:    2.29.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.6.3
Revision:       857c6c6a6a9
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.9
URL:            http://as-git.autostoresystem.com
HTTP Clone URL: http://as-git.autostoresystem.com/some-group/some-project.git
SSH Clone URL:  git@as-git.autostoresystem.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.13.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

    Technical proposal

    We validate jobs dependencies but we look only at the expire_at timestamp. And we also need to include the pipeline.artifacts_locked? check when exposing the job payload to the runner:

    diff --git a/app/models/ci/build.rb b/app/models/ci/build.rb
index 1f7f5f6babe..410af1ecb55 100644
--- a/app/models/ci/build.rb
+++ b/app/models/ci/build.rb
@@ -845,7 +845,7 @@ def invalid_dependencies
     end
 
     def valid_dependency?
-      return false if artifacts_expired?
+      return false if artifacts_expired? && !pipeline.artifacts_locked?
       return false if erased?
 
       true
diff --git a/lib/api/entities/job_request/dependency.rb b/lib/api/entities/job_request/dependency.rb
index 7d6ec832ba1..9af4d2e2550 100644
--- a/lib/api/entities/job_request/dependency.rb
+++ b/lib/api/entities/job_request/dependency.rb
@@ -5,7 +5,7 @@ module Entities
     module JobRequest
       class Dependency < Grape::Entity
         expose :id, :name, :token
-        expose :artifacts_file, using: Entities::Ci::JobArtifactFile, if: ->(job, _) { job.artifacts? }
+        expose :artifacts_file, using: Entities::Ci::JobArtifactFile, if: ->(job, _) { job.available_artifacts? }
       end
     end
   end
    Edited by Marius Bobin

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply