Artifact upload fails with timeout awaiting response headers
Summary
Runner times out when uploading large artifacts to coordinator.
Steps to reproduce
- Confirming whether it can be reproduced on gitlab.com, but seems like a simple (very) large artifact upload can trigger a timeout.
.gitlab-ci.yml
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: never
- if: $CI_COMMIT_BRANCHstages:
- .pre
- build
- test
- deploy
- uploadinclude:
- project: 'redacted'
ref: main
file: '/ci-templates/setup-job.yml' - project: 'redacted'
ref: main
file: '/ci-templates/build-job.yml' - project: 'redacted'
ref: main
file: '/ci-templates/clean-job.yml' - project: 'redacted'
ref: main
file: '/ci-templates/envvar-job.yml'variables:
# CI_DEBUG_TRACE: "true"
BLINX_PRODUCT: "blue"
MULTICONFIG: "mc:x86_64:"
ARCH: "qemux86-64"
PRODUCT_EXT: ".ova"
ARCH_ALT: "x86_64"
BUILD_TARGET: "blue-idk-ova-image"
VERSION: "0001"
YOCTO_CODENAME: "redacted"
RUNNER_TAG: "dev-v1-EC2"
DEPLOY_DIR: "${HOME_DIR}/build/blinx-build/deploy-blinx/images/${ARCH}"salt_upload:
variables:
CI_DEBUG_TRACE: "true"
SALT_PROJECT_ID: 21
image: redacted/curlimages/curl:8.00.1
stage: upload
script:
- 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file dev-ec2 "${CI_API_V4_URL}/projects/${SALT_PROJECT_ID}/redacted"'
- 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file build-env-setup "${CI_API_V4_URL}/projects/${SALT_PROJECT_ID}/redacted"'
- 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ec2-bashrc "${CI_API_V4_URL}/projects/${SALT_PROJECT_ID}/redacted"'
rules:
- if: $CI_COMMIT_BRANCH == "main"
changes:
- dev-ec2
- build-env-setup
- ec2-bashrc
when: always
- when: neverblue-core:
stage: build
tags:
- ${RUNNER_TAG}
extends:
- .setup-env
- .common-clean-job
- .common-build
- .print-common-vars before_script:
- !reference [.common-clean-job, before_script]
- !reference [.print-common-vars, before_script]
script:
- !reference [.setup-env, script]
- !reference [.common-build, script] after_script:
- !reference [.print-common-vars, after_script] artifacts:
name: "${redacted}"
paths:
- ${ARTIFACTS_DIR}/*.tgz
expire_in: 2 weeks
Actual behavior
Job fails with a timeout when uploading artifacts to the coordinator.
Expected behavior
Timeout should be configurable to accomodate larger files.
Relevant logs and/or screenshots
Backend logs
{
"content_type": "text/plain; charset=utf-8",
"correlation_id": "01HEXPM09C7JV66JRGDPSKVSEX",
"duration_ms": 600403,
"host": "sanitized",
"level": "info",
"method": "POST",
"msg": "access",
"proto": "HTTP/1.1",
"referrer": "",
"remote_addr": "",
"remote_ip": "10.41.22.174",
"route": "^/api/v4/jobs/[0-9]+/artifacts\\z",
"status": 500,
"system": "http",
"time": "2023-11-10T22:56:08Z",
"ttfb_ms": 600403,
"uri": "/api/v4/jobs/914660/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks",
"user_agent": "gitlab-runner 16.5.0 (16-5-stable; go1.20.10; linux/amd64)",
"written_bytes": 22
}
{
"correlation_id": "01HEXPM09C7JV66JRGDPSKVSEX",
"error": "handleFileUploads: extract files from multipart: persisting multipart file: context canceled",
"level": "error",
"method": "POST",
"msg": "",
"time": "2023-11-10T22:56:08Z",
"uri": "/api/v4/jobs/914660/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks"
}
Job logs
Uploading artifacts...
5167Runtime platform arch=amd64 os=linux pid=3644423 revision=853330f9 version=16.5.0
redacted/*.tar.gz: found 1 matching artifact files and directories
5169ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks: Post "https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks": net/http: timeout awaiting response headers id=751341 token=64_dwdKy
5170WARNING: Retrying... context=artifacts-uploader error=invalid argument
5171ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks: Post "https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks": net/http: timeout awaiting response headers id=751341 token=64_dwdKy
5172WARNING: Retrying... context=artifacts-uploader error=invalid argument
5173ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks: Post "https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks": net/http: timeout awaiting response headers id=751341 token=64_dwdKy
5174FATAL: invalid argument
Environment description
Helm chart deployment (up to 2,000)
config.toml contents
listen_address = ":9252"
concurrent = 1
check_interval = 3
log_format = "json"
shutdown_timeout = 0
[session_server]
session_timeout = 0
[[runners]]
output_limit = 1000000
name = "Shell executor on EC2 with i-<redacted>"
url = "https://gitlab.redacted"
id = 1515
token = "<redacted>"
token_obtained_at = 2023-11-30T19:56:04Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "shell"
[runners.cache]
MaxUploadedArchiveSize = 0
Used GitLab Runner version
0.58.1
gitlab-runner --version
Version: 16.6.1
Git revision: f5da3c5a
Git branch: 16-6-stable
GO version: go1.20.10
Built: 2023-11-24T21:11:36+0000
OS/Arch: linux/amd64