Support fetching last successful from Artifacts Archive with API

Summary

Getting 404s from Download the artifacts archive API endpoint (both in CI jobs and from my workstation) after n other jobs (not the one we're querying for) have also run on the project. It seems to correlate to when the job we're looking for is no longer on the first page of the UI.

It would be great if the Artifacts Archive API endpoint would fetch the latest matched artifact for a given job name + :ref, even if it wasn't the very last job run.

Since this endpoint provides such great functionality that it does already exist in the Gitlab API, I probably don't need to convince you this is a valuable capability. However, how it currently works would limit us to only having a single job for a given repository, and it would also break if there was a failure (from what I've gathered from the other ticket).

Steps to reproduce

  1. Create a job (test job) that tries to fetch the artifact from its own previous job run using Download the artifacts archive
  2. Run a different job (or jobs - control job(s)) enough times that the job you're testing is well into the second page of the the Artifacts UI
  3. Run the "test job" again

The Download the artifacts archive call will work until enough other jobs run that the "test job" latest run (on the artifacts page) doesn't appear until somewhere down page 2.

Example Project

I've recreated it in this public project https://gitlab.com/mschencksap1/mschencksap

I've tried it both with JOB-TOKEN and PRIVATE-TOKEN auth headers. The JOB-TOKEN approach doesn't work but I expect that since this is a free-tier account.

You can see a successful run here (4 runs in a row showing the accumulated history):

Then I ran the "filler" stage 3 times, then the "test" stage again: https://gitlab.com/mschencksap1/mschencksap/-/jobs

The very next run of the "test" stage shows it 404ing:

What's also odd is that the call fails when I just remove the (not working) testjobtoken from the .gitlab-ci.yml. So I'm really not sure what's going on but this seems really flakey.

What is the current bug behavior?

404ing when the artifact is still available (even when I set expire_in to several weeks. The 404ing behavior can occur as soon as 10 minutes after the "test job" completes.

What is the expected correct behavior?

I expect to still be able to fetch the latest artifact of the given job, as long as the artifact hasn't expired. Just like I can fetch it via get artifact by job id

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:env:info\`) (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 

(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`)

(For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`)

(we will only investigate if the tests are passing)

Possible fixes

Edited by Rutvik Shah