Download latest artifacts in CI job

Problem to solve

GitLab's "latest job artifacts" API endpoint only returns artifacts from completed pipelines, making artifacts unavailable to subsequent stages within the same running pipeline.

Expected Behavior: Users expect to be able to:

  • Build stage: Create and upload artifacts
  • Deploy stage: Download those artifacts using the "latest" artifacts API within the same pipeline

Actual Behavior:

  • Artifacts from the current (running) pipeline return HTTP 404
  • Only artifacts from previously completed pipelines are accessible via the "latest" endpoint
  • Forces users to split work across multiple pipeline runs instead of stages

Impact:

  • Broken workflow: Cannot share artifacts between stages in a single pipeline

  • Inefficient process: Requires running pipeline twice:

    • First pipeline: Build and upload artifacts
    • Second pipeline: Download artifacts and deploy
  • Workaround limitation: Manual wget downloads work, but GitLab's built-in artifact sharing doesn't

Root Cause: The "latest job artifacts" endpoint definition treats "latest" as "from the most recent completed pipeline" rather than "most recent available artifacts including current pipeline."

User Need: Access to artifacts from earlier stages within the same pipeline execution, enabling proper CI/CD stage progression without artificial pipeline splitting. This prevents the standard CI/CD pattern of build → test → deploy within a single pipeline executio

Summary from 2017-08-02

From the doc here,we can download the latest job artifacts by CI_PROJECT_NAMESPACE,CI_PROJECT_NAME,CI_COMMIT_REF_NAME , this is useful when i need shared artifacts between stages.

In my understanding the latest job artifacts is the newest one on current branch. but, the fact is : artifacts by latest pipeline run. that means I can not run CI stages like:

  1. build - build project,upload artifacts
  2. deploy - retrieve artifacts and deploy(e.g. push to docker registry)

I have to run pipeline twice(see screen pic below):

  1. do build in 1st pipeline
  2. do deploy in 2nd pipeline

I am not sure this is a bug or feature-request,I need shared artifacts between stages.

Steps to reproduce

  1. run job to upload artifacts
  2. run job to download artifacts (404 error)

Use wet to download it in shell,it works.

Actual behavior

artifacts not found(http status 404)

Expected behavior

download artifacts successfully

Relevant logs and/or screenshots

example project : https://gitlab.com/cj216/dl-latest-artifacts

.gitlab-ci.yml

variables:

  HELLO_FILE: "web/Hello.html"
  # see https://docs.gitlab.com/ce/user/project/pipelines/job_artifacts.html#downloading-the-latest-job-artifacts
  LAST_UPLOADED_FILE: "http://192.168.0.221/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}/builds/artifacts/${CI_COMMIT_REF_NAME}/raw/${HELLO_FILE}?job=html-artifacts"

stages:
  - build 
  - test 

html-artifacts:
  stage: build
  when: manual
  script:
    - gitlab-runner --version
  artifacts:
    name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    paths:
      - ${HELLO_FILE}
    expire_in: 2 hour
  tags:
    - linux-shell

test-html-artifacts:
  stage: test
  when: manual
  dependencies:
    - html-artifacts
  script:
    - mkdir build
    - wget  -S -O build/download-test-$CI_COMMIT_REF_SLUG.html $LAST_UPLOADED_FILE
  #only:
  #  - master
  tags:
    - linux-shell

html-artifacts output

Running with gitlab-ci-multi-runner 9.4.1 (d24b11c)
  on linux-shell (b71ac640)
Using Shell executor...
Running on linux-ci-1...
Cloning repository...
Cloning into '/home/gitlab-runner/builds/b71ac640/0/chenjun/test-latest-artifacts'...
Checking out 2ba897e7 as master...
Skipping Git submodules setup
$ gitlab-runner --version
Version:      9.4.1
Git revision: d24b11c
Git branch:   9-4-stable
GO version:   go1.8.3
Built:        Tue, 25 Jul 2017 12:04:47 +0000
OS/Arch:      linux/amd64
Uploading artifacts...
web/Hello.html: found 1 matching files             
Uploading artifacts to coordinator... ok            id=1514 responseStatus=201 Created token=yB3XwiuQ
Job succeeded

test-html-artifacts output

Skipping Git submodules setup
Downloading artifacts for html-artifacts (1514)...
Downloading artifacts from coordinator... ok        id=1514 responseStatus=200 OK token=yB3XwiuQ
$ mkdir build
$ wget  -S -O build/download-test-$CI_COMMIT_REF_SLUG.html $LAST_UPLOADED_FILE
--2017-08-03 10:31:02--  http://192.168.0.221/chenjun/test-latest-artifacts/builds/artifacts/master/raw/web/Hello.html?job=html-artifacts
Connecting to 192.168.0.221:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 404 Not Found
  Server: nginx
  Date: Thu, 03 Aug 2017 02:31:02 GMT
  Content-Type: text/html; charset=utf-8
  Content-Length: 2943
  Connection: keep-alive
  Cache-Control: no-cache, no-store, max-age=0, must-revalidate
  Expires: Fri, 01 Jan 1990 00:00:00 GMT
  Pragma: no-cache
  X-Content-Type-Options: nosniff
  X-Frame-Options: DENY
  X-Request-Id: 22d64eb6-e875-4395-9b9f-85b758701df6
  X-Runtime: 0.016654
  X-Ua-Compatible: IE=edge
  X-Xss-Protection: 1; mode=block
2017-08-03 10:31:02 ERROR 404: Not Found.

ERROR: Job failed: exit status 1

pipeline screen captures

ci-pipeline-screen

Environment description

  • gitlab-ce : 9.4.3
  • gitlab ci runner: 9.4.1 (shell executor,as shared runner)

Used GitLab Runner version

zfmm@linux-ci-1:~/temp$ gitlab-runner --version
Version:      9.4.1
Git revision: d24b11c
Git branch:   9-4-stable
GO version:   go1.8.3
Built:        Tue, 25 Jul 2017 12:04:47 +0000
OS/Arch:      linux/amd64
Edited by Darren Eastman