Download latest artifacts in CI job
Problem to solve
GitLab's "latest job artifacts" API endpoint only returns artifacts from completed pipelines, making artifacts unavailable to subsequent stages within the same running pipeline.
Expected Behavior: Users expect to be able to:
- Build stage: Create and upload artifacts
- Deploy stage: Download those artifacts using the "latest" artifacts API within the same pipeline
Actual Behavior:
- Artifacts from the current (running) pipeline return HTTP 404
- Only artifacts from previously completed pipelines are accessible via the "latest" endpoint
- Forces users to split work across multiple pipeline runs instead of stages
Impact:
-
Broken workflow: Cannot share artifacts between stages in a single pipeline
-
Inefficient process: Requires running pipeline twice:
- First pipeline: Build and upload artifacts
- Second pipeline: Download artifacts and deploy
-
Workaround limitation: Manual wget downloads work, but GitLab's built-in artifact sharing doesn't
Root Cause: The "latest job artifacts" endpoint definition treats "latest" as "from the most recent completed pipeline" rather than "most recent available artifacts including current pipeline."
User Need: Access to artifacts from earlier stages within the same pipeline execution, enabling proper CI/CD stage progression without artificial pipeline splitting. This prevents the standard CI/CD pattern of build → test → deploy within a single pipeline executio
Summary from 2017-08-02
From the doc here,we can download the latest job artifacts by CI_PROJECT_NAMESPACE,CI_PROJECT_NAME,CI_COMMIT_REF_NAME , this is useful when i need shared artifacts between stages.
In my understanding the latest job artifacts is the newest one on current branch. but, the fact is : artifacts by latest pipeline run. that means I can not run CI stages like:
- build - build project,upload artifacts
- deploy - retrieve artifacts and deploy(e.g. push to docker registry)
I have to run pipeline twice(see screen pic below):
- do build in 1st pipeline
- do deploy in 2nd pipeline
I am not sure this is a bug or feature-request,I need shared artifacts between stages.
Steps to reproduce
- run job to upload artifacts
- run job to download artifacts (404 error)
Use wet to download it in shell,it works.
Actual behavior
artifacts not found(http status 404)
Expected behavior
download artifacts successfully
Relevant logs and/or screenshots
example project : https://gitlab.com/cj216/dl-latest-artifacts
.gitlab-ci.yml
variables:
HELLO_FILE: "web/Hello.html"
# see https://docs.gitlab.com/ce/user/project/pipelines/job_artifacts.html#downloading-the-latest-job-artifacts
LAST_UPLOADED_FILE: "http://192.168.0.221/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}/builds/artifacts/${CI_COMMIT_REF_NAME}/raw/${HELLO_FILE}?job=html-artifacts"
stages:
- build
- test
html-artifacts:
stage: build
when: manual
script:
- gitlab-runner --version
artifacts:
name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
paths:
- ${HELLO_FILE}
expire_in: 2 hour
tags:
- linux-shell
test-html-artifacts:
stage: test
when: manual
dependencies:
- html-artifacts
script:
- mkdir build
- wget -S -O build/download-test-$CI_COMMIT_REF_SLUG.html $LAST_UPLOADED_FILE
#only:
# - master
tags:
- linux-shellhtml-artifacts output
Running with gitlab-ci-multi-runner 9.4.1 (d24b11c)
on linux-shell (b71ac640)
Using Shell executor...
Running on linux-ci-1...
Cloning repository...
Cloning into '/home/gitlab-runner/builds/b71ac640/0/chenjun/test-latest-artifacts'...
Checking out 2ba897e7 as master...
Skipping Git submodules setup
$ gitlab-runner --version
Version: 9.4.1
Git revision: d24b11c
Git branch: 9-4-stable
GO version: go1.8.3
Built: Tue, 25 Jul 2017 12:04:47 +0000
OS/Arch: linux/amd64
Uploading artifacts...
web/Hello.html: found 1 matching files
Uploading artifacts to coordinator... ok id=1514 responseStatus=201 Created token=yB3XwiuQ
Job succeededtest-html-artifacts output
Skipping Git submodules setup
Downloading artifacts for html-artifacts (1514)...
Downloading artifacts from coordinator... ok id=1514 responseStatus=200 OK token=yB3XwiuQ
$ mkdir build
$ wget -S -O build/download-test-$CI_COMMIT_REF_SLUG.html $LAST_UPLOADED_FILE
--2017-08-03 10:31:02-- http://192.168.0.221/chenjun/test-latest-artifacts/builds/artifacts/master/raw/web/Hello.html?job=html-artifacts
Connecting to 192.168.0.221:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 404 Not Found
Server: nginx
Date: Thu, 03 Aug 2017 02:31:02 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2943
Connection: keep-alive
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Request-Id: 22d64eb6-e875-4395-9b9f-85b758701df6
X-Runtime: 0.016654
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block
2017-08-03 10:31:02 ERROR 404: Not Found.
ERROR: Job failed: exit status 1pipeline screen captures
Environment description
gitlab-ce: 9.4.3gitlab ci runner: 9.4.1 (shell executor,as shared runner)
Used GitLab Runner version
zfmm@linux-ci-1:~/temp$ gitlab-runner --version
Version: 9.4.1
Git revision: d24b11c
Git branch: 9-4-stable
GO version: go1.8.3
Built: Tue, 25 Jul 2017 12:04:47 +0000
OS/Arch: linux/amd64