Uploading artifacts as "archive" to coordinator failed after upgrade from 14.0.5 to 14.1.2

Summary

The artifact upload feature of gitlab CI no longer works after upgrading gitlab-ee (14.0.5 => 14.1.2) and gitlab-runner (14.0.1 => 14.1.0). We haven't had any trouble with this feature until this specific upgrade.

We have found these similar (below), but have found no workarounds or other helpful information. No other issues appear to reference this failure in relation to a version upgrade.

We use AWS s3 for storage and none of the artifacts are large.

Steps to reproduce

Perform the upgrade as described above.

Example Project

What is the current bug behavior?

All jobs which use artifact storage are now failing with the following message:

Uploading artifacts...
target/scala-2.13/scoverage-report/: found 27 matching files and directories 
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=40177 responseStatus=500 Internal Server Error status=500 token=Q_aDmbhv
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=40177 responseStatus=500 Internal Server Error status=500 token=Q_aDmbhv
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=40177 responseStatus=500 Internal Server Error status=500 token=Q_aDmbhv
FATAL: invalid argument                            
Cleaning up file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

What is the expected correct behavior?

Artifacts should be uploaded without failure.

Relevant logs and/or screenshots

I was able to find some requests in the logs (redacted): /var/log/gitlab/gitlab-workhorse/current:

{
    "content_type": "application/json",
    "correlation_id": "01FD9N2V0KM38PKK7H6XXT0B8J",
    "duration_ms": 560,
    "host": "gitlab.example.com",
    "level": "info",
    "method": "POST",
    "msg": "access",
    "proto": "HTTP/1.1",
    "referrer": "",
    "remote_addr": "0.0.0.0:0",
    "remote_ip": "0.0.0.0",
    "route": "^/api/v4/jobs/[0-9]+/artifacts\\z",
    "status": 500,
    "system": "http",
    "time": "2021-08-17T08:54:10Z",
    "ttfb_ms": 560,
    "uri": "/api/v4/jobs/40170/artifacts?artifact_format=zip\u0026artifact_type=archive",
    "user_agent": "gitlab-runner 14.1.0-rc1 (14-1-stable; go1.13.8; linux/amd64)",
    "written_bytes": 26
}

/var/log/gitlab/gitlab-rails/api.json:

{
    "time": "2021-08-17T08:17:21.893Z",
    "severity": "INFO",
    "duration_s": 0.20832,
    "db_duration_s": 0.0178,
    "view_duration_s": 0.19052,
    "status": 500,
    "method": "POST",
    "path": "/api/v4/jobs/40166/artifacts",
    "params": [
        {
            "key": "artifact_format",
            "value": "zip"
        },
        {
            "key": "artifact_type",
            "value": "archive"
        },
        {
            "key": "file.name",
            "value": "artifacts.zip"
        },
        {
            "key": "file.remote_id",
            "value": ""
        },
        {
            "key": "file.md5",
            "value": ""
        },
        {
            "key": "file.sha256",
            "value": ""
        },
        {
            "key": "file.sha512",
            "value": ""
        },
        {
            "key": "file.gitlab-workhorse-upload",
            "value": ""
        },
        {
            "key": "file.size",
            "value": "5242880"
        },
        {
            "key": "file.path",
            "value": ""
        },
        {
            "key": "file.sha1",
            "value": ""
        },
        {
            "key": "metadata.remote_url",
            "value": ""
        },
        {
            "key": "metadata.remote_id",
            "value": ""
        },
        {
            "key": "metadata.sha1",
            "value": ""
        },
        {
            "key": "metadata.sha512",
            "value": ""
        },
        {
            "key": "metadata.name",
            "value": "metadata.gz"
        },
        {
            "key": "metadata.size",
            "value": "0"
        },
        {
            "key": "metadata.md5",
            "value": ""
        },
        {
            "key": "metadata.sha256",
            "value": ""
        },
        {
            "key": "metadata.gitlab-workhorse-upload",
            "value": ""
        },
        {
            "key": "metadata.path",
            "value": "/tmp/metadata.gz827707957"
        },
        {
            "key": "file",
            "value": null
        },
        {
            "key": "metadata",
            "value": null
        }
    ],
    "host": "gitlab.example.com",
    "remote_ip": "0.0.0.0, 0.0.0.0",
    "ua": "gitlab-runner 14.1.0-rc1 (14-1-stable; go1.13.8; linux/amd64)",
    "route": "/api/:version/jobs/:id/artifacts",
    "queue_duration_s": 0.024788,
    "redis_calls": 1,
    "redis_duration_s": 0.000198,
    "redis_read_bytes": 2,
    "redis_write_bytes": 114,
    "redis_cache_calls": 1,
    "redis_cache_duration_s": 0.000198,
    "redis_cache_read_bytes": 2,
    "redis_cache_write_bytes": 114,
    "db_count": 11,
    "db_write_count": 0,
    "db_cached_count": 0,
    "external_http_count": 4,
    "external_http_duration_s": 0.03541821491671726,
    "cpu_s": 0.052433,
    "mem_objects": 30305,
    "mem_bytes": 3519976,
    "mem_mallocs": 14198,
    "mem_total_bytes": 4732176,
    "correlation_id": "01FD9JZD0GZGZNBS3TQ89S5GQP",
    "meta.user": "",
    "meta.project": "",
    "meta.root_namespace": "",
    "meta.subscription_plan": "",
    "meta.caller_id": "POST /api/:version/jobs/:id/artifacts",
    "meta.remote_ip": "0.0.0.0",
    "meta.feature_category": "runner",
    "meta.client_id": "user/18",
    "content_length": "8186"
}

/var/log/gitlab/gitlab-rails/production.log:

Started POST "/api/v4/jobs/40172/artifacts/authorize?artifact_format=zip&artifact_type=archive" for 0.0.0.0 at 2021-08-17 09:08:52 +0000
Started POST "/api/v4/jobs/40172/artifacts?artifact_format=zip&artifact_type=archive" for 0.0.0.0 at 2021-08-17 09:08:53 +0000

There were no exceptions written to /var/log/gitlab/gitlab-rails/exceptions_json.log.

Output of checks

Results of GitLab environment info

System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.3 Redis Version: 6.0.14 Git Version: 2.32.0 Sidekiq Version:5.2.9 Go Version: unknown

GitLab information Version: 14.1.2-ee Revision: 0bf9b154 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.6 URL: https://gitlab.example.com HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: no

GitLab Shell Version: 13.19.1 Repository storage paths:

  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Checking Reply by email ...

IMAP server credentials are correct? ... Checking example-gitlab@example.com yes Init.d configured correctly? ... skipped MailRoom running? ... skipped

Checking Reply by email ... Finished

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.31.0 ? ... yes (2.32.0) Git user has default SSH configuration? ... yes Active users: ... 378 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes