Artifact upload fails with timeout awaiting response headers

Summary

Runner times out when uploading large artifacts to coordinator.

Steps to reproduce

  • Confirming whether it can be reproduced on gitlab.com, but seems like a simple (very) large artifact upload can trigger a timeout.
.gitlab-ci.yml 
workflow:

  rules:

    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

      when: never

    - if: $CI_COMMIT_BRANCHstages:

  - .pre

  - build

  - test

  - deploy

  - uploadinclude:

  - project: 'redacted'

    ref: main

    file: '/ci-templates/setup-job.yml'  - project: 'redacted'

    ref: main

    file: '/ci-templates/build-job.yml'  - project: 'redacted'

    ref: main

    file: '/ci-templates/clean-job.yml'  - project: 'redacted'

    ref: main

    file: '/ci-templates/envvar-job.yml'variables:

#  CI_DEBUG_TRACE: "true"

  BLINX_PRODUCT: "blue"

  MULTICONFIG: "mc:x86_64:"

  ARCH: "qemux86-64"

  PRODUCT_EXT: ".ova"

  ARCH_ALT: "x86_64"

  BUILD_TARGET: "blue-idk-ova-image"

  VERSION: "0001"

  YOCTO_CODENAME: "redacted"

  RUNNER_TAG: "dev-v1-EC2"

  DEPLOY_DIR: "${HOME_DIR}/build/blinx-build/deploy-blinx/images/${ARCH}"salt_upload:

  variables:

    CI_DEBUG_TRACE: "true"

    SALT_PROJECT_ID: 21

  image: redacted/curlimages/curl:8.00.1

  stage: upload

  script:

    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file dev-ec2 "${CI_API_V4_URL}/projects/${SALT_PROJECT_ID}/redacted"'

    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file build-env-setup "${CI_API_V4_URL}/projects/${SALT_PROJECT_ID}/redacted"'

    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ec2-bashrc "${CI_API_V4_URL}/projects/${SALT_PROJECT_ID}/redacted"'

  rules:

    - if: $CI_COMMIT_BRANCH == "main"

      changes:

        - dev-ec2

        - build-env-setup

        - ec2-bashrc

      when: always

    - when: neverblue-core:

  stage: build

  tags:

  - ${RUNNER_TAG}

  extends:

    - .setup-env

    - .common-clean-job

    - .common-build

    - .print-common-vars  before_script:

    - !reference [.common-clean-job, before_script]

    - !reference [.print-common-vars, before_script]

  script:

    - !reference [.setup-env, script]

    - !reference [.common-build, script]  after_script:

    - !reference [.print-common-vars, after_script]  artifacts:

    name: "${redacted}"

    paths:

      - ${ARTIFACTS_DIR}/*.tgz

    expire_in: 2 weeks

Actual behavior

Job fails with a timeout when uploading artifacts to the coordinator.

Expected behavior

Timeout should be configurable to accomodate larger files.

Relevant logs and/or screenshots

Backend logs 

{
  "content_type": "text/plain; charset=utf-8",
  "correlation_id": "01HEXPM09C7JV66JRGDPSKVSEX",
  "duration_ms": 600403,
  "host": "sanitized",
  "level": "info",
  "method": "POST",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "",
  "remote_addr": "",
  "remote_ip": "10.41.22.174",
  "route": "^/api/v4/jobs/[0-9]+/artifacts\\z",
  "status": 500,
  "system": "http",
  "time": "2023-11-10T22:56:08Z",
  "ttfb_ms": 600403,
  "uri": "/api/v4/jobs/914660/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks",
  "user_agent": "gitlab-runner 16.5.0 (16-5-stable; go1.20.10; linux/amd64)",
  "written_bytes": 22
}

{
  "correlation_id": "01HEXPM09C7JV66JRGDPSKVSEX",
  "error": "handleFileUploads: extract files from multipart: persisting multipart file: context canceled",
  "level": "error",
  "method": "POST",
  "msg": "",
  "time": "2023-11-10T22:56:08Z",
  "uri": "/api/v4/jobs/914660/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks"
}

Job logs 
Uploading artifacts...

5167Runtime platform  arch=amd64 os=linux pid=3644423 revision=853330f9 version=16.5.0

redacted/*.tar.gz: found 1 matching artifact files and directories

5169ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks: Post "https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks": net/http: timeout awaiting response headers id=751341 token=64_dwdKy

5170WARNING: Retrying...  context=artifacts-uploader error=invalid argument

5171ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks: Post "https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks": net/http: timeout awaiting response headers id=751341 token=64_dwdKy

5172WARNING: Retrying...  context=artifacts-uploader error=invalid argument

5173ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks: Post "https://redacted/api/v4/jobs/751341/artifacts?artifact_format=zip&artifact_type=archive&expire_in=2+weeks": net/http: timeout awaiting response headers id=751341 token=64_dwdKy

5174FATAL: invalid argument

Environment description

Helm chart deployment (up to 2,000)

config.toml contents 
listen_address = ":9252"
concurrent = 1
check_interval = 3
log_format = "json"
shutdown_timeout = 0

[session_server]
  session_timeout = 0

[[runners]]
  output_limit = 1000000
  name = "Shell executor on EC2 with i-<redacted>"
  url = "https://gitlab.redacted"
  id = 1515
  token = "<redacted>"
  token_obtained_at = 2023-11-30T19:56:04Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  [runners.cache]
    MaxUploadedArchiveSize = 0

Used GitLab Runner version

0.58.1

gitlab-runner --version 


Version:      16.6.1

Git revision: f5da3c5a

Git branch:   16-6-stable

GO version:   go1.20.10

Built:        2023-11-24T21:11:36+0000

OS/Arch:      linux/amd64

Possible fixes

ZD Ticket (Internal)

https://gitlab.zendesk.com/agent/tickets/470164

Edited by Daniel Diniz