Job in `needs` ignored if latest but expired

Summary

When using needs to retrieve a cross-project artifact (see https://docs.gitlab.com/ee/ci/yaml/#cross-project-artifact-downloads-with-needs), GitLab will completely ignore the needs entry once the target artifact is expired but still kept because it is the latest.

I.o.w., if the target artifact is forcibly kept because it is the latest for the given ref but it is also expired, it is as if it wasn't even in the needs array at all.

EDIT: It turns out this is not limited to cross-project ones but also within the same project and pipeline.

Steps to reproduce

1. Create a project with a pipeline that generates an artifact.
2. Run said pipeline and make sure the arifact is created and GitLab tells you that the artifact will be kept regardless of expiration due to it being the latest for the given ref.
3. Make a second project with a pipeline that has a job that needs the previously created project and artifacts.
4. Run the second pipeline AFTER the artifact from the first pipeline has become expired.
5. Check the job logs from GitLab to see it doesn't even attempt to download the artifact and just continues with the job.

Example Project

https://gitlab.com/Omar007/gitlab-issue-271261/

What is the current bug behavior?

The GitLab CI pipeline process just completely ignores the cross-project needs entry as if it wasn't even defined in the needs array at all.

What is the expected correct behavior?

It should ALWAYS download the latest artifact if it is present, regardless of it being expired or not.

If you do not agree with that statement and this was on purpose, GitLab should at the very least error out when resolving the artifacts with an error along the lines of "The available artifact is expired." instead of silently ignoring the whole dependency.

Relevant logs and/or screenshots

When the artifact is latest and/or not expired

Downloading artifacts
Downloading artifacts for job1a (54426)...
Downloading artifacts from coordinator... ok        id=54426 responseStatus=200 OK token=Chehfszd
Downloading artifacts for job2a (57974)...
Downloading artifacts from coordinator... ok        id=57974 responseStatus=200 OK token=BGasRFqU
Downloading artifacts for job2b (57975)...
Downloading artifacts from coordinator... ok        id=57975 responseStatus=200 OK token=-whyXXhz
Executing "step_script" stage of the job script

When the artifact is latest but expired

Downloading artifacts
Downloading artifacts for job2a (57974)...
Downloading artifacts from coordinator... ok        id=57974 responseStatus=200 OK token=BGasRFqU
Downloading artifacts for job2b (57975)...
Downloading artifacts from coordinator... ok        id=57975 responseStatus=200 OK token=-whyXXhz
Executing "step_script" stage of the job script

With this last one it just completely ignored the 'job1a' dependency/artifacts (the bug).

Relevant needs array:

  needs:
    - job: job2a
      artifacts: true
    - job: job2b
      artifacts: true
    - project: some-group/first-project
      ref: master
      job: job1a
      artifacts: true

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.6.6p146
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.9
Git Version:    2.28.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.4.1-ee
Revision:       4b9c8135cd9
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.9
URL:            x
HTTP Clone URL: x/some-group/some-project.git
SSH Clone URL:  x/some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.7.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.7.0 ? ... OK (13.7.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 15 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
4/2 ... yes
89/3 ... yes
89/4 ... yes
89/5 ... yes
89/6 ... yes
80/7 ... yes
80/8 ... yes
80/9 ... yes
80/10 ... yes
80/11 ... yes
80/12 ... yes
80/13 ... yes
3/14 ... yes
3/15 ... yes
72/16 ... yes
72/17 ... yes
64/18 ... yes
5/20 ... yes
5/21 ... yes
3/23 ... yes
3/24 ... yes
5/25 ... yes
5/26 ... yes
3/27 ... yes
5/28 ... yes
3/29 ... yes
21/30 ... yes
21/31 ... yes
21/32 ... yes
3/33 ... yes
21/34 ... yes
88/35 ... yes
21/37 ... yes
21/38 ... yes
21/39 ... yes
21/40 ... yes
89/41 ... yes
88/42 ... yes
89/47 ... yes
89/51 ... yes
89/53 ... yes
89/54 ... yes
89/56 ... yes
89/57 ... yes
89/58 ... yes
89/59 ... yes
89/60 ... yes
89/61 ... yes
89/62 ... yes
89/63 ... yes
89/64 ... yes
89/65 ... yes
89/66 ... yes
89/67 ... yes
89/68 ... yes
89/69 ... yes
89/70 ... yes
89/71 ... yes
89/72 ... yes
89/73 ... yes
89/74 ... yes
89/75 ... yes
89/76 ... yes
89/77 ... yes
89/78 ... yes
89/79 ... yes
89/82 ... yes
5/83 ... yes
3/85 ... yes
63/86 ... yes
64/87 ... yes
64/88 ... yes
64/89 ... yes
64/90 ... yes
88/91 ... yes
3/93 ... yes
65/94 ... yes
65/95 ... yes
65/96 ... yes
64/97 ... yes
64/98 ... yes
89/99 ... yes
24/100 ... yes
89/102 ... yes
89/105 ... yes
17/106 ... yes
71/108 ... yes
72/109 ... yes
3/111 ... yes
89/114 ... yes
89/118 ... yes
76/119 ... yes
89/124 ... yes
25/125 ... yes
78/126 ... yes
78/127 ... yes
78/128 ... yes
78/129 ... yes
88/131 ... yes
89/132 ... yes
12/134 ... yes
79/135 ... yes
88/136 ... yes
18/141 ... yes
18/142 ... yes
88/143 ... yes
79/144 ... yes
89/146 ... yes
76/147 ... yes
89/149 ... yes
76/150 ... yes
24/152 ... yes
18/153 ... yes
18/154 ... yes
24/156 ... yes
89/158 ... yes
24/159 ... yes
85/160 ... yes
87/161 ... yes
87/162 ... yes
87/163 ... yes
71/165 ... yes
12/166 ... yes
85/167 ... yes
76/168 ... yes
90/170 ... yes
90/171 ... yes
70/172 ... yes
89/173 ... yes
24/174 ... yes
24/175 ... yes
18/178 ... yes
81/179 ... yes
70/180 ... yes
85/185 ... yes
12/190 ... yes
89/191 ... yes
81/192 ... yes
71/195 ... yes
71/198 ... yes
94/199 ... yes
94/200 ... yes
94/201 ... yes
94/202 ... yes
85/203 ... yes
78/207 ... yes
95/210 ... yes
95/211 ... yes
90/212 ... yes
93/213 ... yes
96/214 ... yes
97/215 ... yes
91/216 ... yes
90/217 ... yes
90/218 ... yes
70/219 ... yes
70/220 ... yes
89/221 ... yes
90/222 ... yes
90/223 ... yes
91/224 ... yes
93/225 ... yes
97/226 ... yes
87/227 ... yes
90/228 ... yes
90/229 ... yes
12/230 ... yes
2/231 ... yes
97/233 ... yes
89/234 ... yes
87/235 ... yes
90/236 ... yes
89/237 ... yes
90/238 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 14
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

We do not have a fix for this but we may have a work-around that seems to work for us; the Keep button makes a change to the artifact record making the needs entry work again.

Once the latest artifact is expired you can click on the Keep button and try to re-run the job that needed the artifact. It should now suddenly decide to grab the artifact again instead of silently ignoring the dependency.
You will have to somehow keep in mind this was done for that artifact as to remove it manually again later once it is no longer the latest one.