Uploading artifacts fails with 500 error

Potential Cause

Please see #444271 (comment 2184142529) as a potential cause. Please leave a comment if that's not the case for your issue so that we can continue investigating.

Description

Self hosted lab via docker.
Lab image: ce:16.8.3-ce.0
Runner: v16.9.1

After upgrading from 15.x to 16.x version pipelines fails with 500 internal error.

WARNING: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error  id=1426 responseStatus=500 Internal Server Error status=500 token=64_LxzFt
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error  id=1426 responseStatus=500 Internal Server Error status=500 token=64_LxzFt
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error  id=1426 responseStatus=500 Internal Server Error status=500 token=64_LxzFt
FATAL: invalid argument                            
ERROR: Job failed: exit code 1

In logs found the following:

==> /var/log/gitlab/gitlab-rails/production.log <==
  
EOFError (EOFError):
  
config/initializers/rack_multipart_patch.rb:10:in `extract_multipart'
lib/gitlab/rack_attack/request.rb:265:in `deprecated_api_request?'
lib/gitlab/rack_attack/request.rb:195:in `throttle_unauthenticated_deprecated_api?'
lib/gitlab/rack_attack/request.rb:95:in `throttle_unauthenticated_api?'
lib/gitlab/rack_attack/request.rb:86:in `throttle?'
lib/gitlab/rack_attack.rb:139:in `block (2 levels) in configure_throttles'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'