Workhorse artifact metadata temp path falls back to os.TempDir() when using object storage direct upload, causing TMPDIR mismatch with Rails

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When object storage with direct upload is enabled and a user configures a custom TMPDIR via gitlab_rails['env'] (as documented in the Omnibus environment variables docs), CI artifact uploads fail with 400 Bad Request and Content-Type: text/plain.

The root cause is in workhorse/internal/upload/artifacts_uploader.go. When Workhorse generates artifact metadata (metadata.gz) from a ZIP artifact, it needs a local temp directory. If the Rails /authorize response provides a RemoteObject (object storage direct upload) rather than a TempPath, the tempDir field is empty and Workhorse falls back to os.TempDir():

func (a *artifactsUploadProcessor) generateMetadataFromZip(...) {
    metaOpts := &destination.UploadOpts{
        LocalTempPath: a.tempDir,
    }
    if metaOpts.LocalTempPath == "" {
        metaOpts.LocalTempPath = os.TempDir()  // Falls back to TMPDIR env var or /tmp
    }
    // ...
}

Rails' multipart middleware (lib/gitlab/middleware/multipart.rb) then validates the metadata temp file path against allowed_paths, which includes Dir.tmpdir (Ruby's equivalent, also respecting TMPDIR). If the user set gitlab_rails['env'] = { 'TMPDIR' => '/var/opt/gitlab/tmp' } but did not set gitlab_workhorse['env'] to match, Dir.tmpdir returns /var/opt/gitlab/tmp while the metadata file is in /tmp. The path validation fails, raising InvalidPathError, which the middleware catches and returns as 400 Bad Request with Content-Type: text/plain:

rescue UploadedFile::InvalidPathError, ApolloUploadServer::GraphQLDataBuilder::OutOfBounds => e
  [400, { 'Content-Type' => 'text/plain' }, [e.message]]
end

This is the same class of issue as #363701 (closed), which was caused by !87255 (merged) and reverted in !90083 (merged). The revert fixed the local storage case by restoring the use of a.tempDir from the Rails authorize response, but the os.TempDir() fallback for the object storage direct upload path was left in place.

Steps to reproduce

  1. Configure a GitLab Omnibus instance with object storage for artifacts (direct upload enabled).
  2. Set a custom TMPDIR for Rails only in /etc/gitlab/gitlab.rb: 
    gitlab_rails['env'] = { 'TMPDIR' => '/var/opt/gitlab/tmp' }
  3. Create the directory with appropriate permissions: 
    sudo mkdir -p /var/opt/gitlab/tmp
sudo chown git:git /var/opt/gitlab/tmp
sudo chmod 700 /var/opt/gitlab/tmp
  4. Run sudo gitlab-ctl reconfigure.
  5. Run a CI job that produces artifacts.

Example Project

N/A - reproducible on any project with CI artifact uploads when the above configuration is applied.

What is the current bug behavior?

Artifact uploads fail with 400 Bad Request. Workhorse logs show:

  • artifacts.zip uploaded successfully via client_mode: "s3_client_v2" (direct to object storage)
  • metadata.gz written as client_mode: "local_tempfile" with local_temp_path: "/tmp"
  • The finalize POST to Rails returns status: 400 with content_type: "text/plain"

What is the expected correct behavior?

Artifact uploads should succeed regardless of whether gitlab_workhorse['env'] has TMPDIR set, because Workhorse should use a temp path that Rails will accept. The metadata temp file should be written to a directory that Rails' multipart middleware considers allowed.

Relevant logs and/or screenshots

Workhorse logs for a failing upload (from a customer report, redacted):

{"client_mode":"s3_client_v2","correlation_id":"...","filename":"artifacts.zip","is_local":false,"is_remote":true,"msg":"Destination: Upload saved file","provider":"AWS"}
{"client_mode":"local_tempfile","correlation_id":"...","filename":"metadata.gz","is_local":true,"is_remote":false,"local_temp_path":"/tmp","msg":"Destination: Upload saved file"}
{"content_type":"text/plain","correlation_id":"...","status":400,"uri":"/api/v4/jobs/.../artifacts?artifact_format=zip&artifact_type=archive"}

Output of checks

N/A - this is a code-level issue, not environment-specific.

Results of GitLab environment info

Reproduced on GitLab 18.8.5 (Omnibus) with object storage (AWS S3, use_iam_profile).

Results of GitLab application Check

N/A

Possible fixes

Several approaches could resolve this:

  1. Have Rails include a metadata temp path in the /authorize response even when using direct upload. Currently, when direct upload is enabled, Rails returns a RemoteObject but no TempPath. Rails could include a TempPath specifically for metadata generation, so Workhorse never needs to fall back to os.TempDir(). This follows the existing design pattern where Rails tells Workhorse where to write.

  2. Add /tmp unconditionally to allowed_paths in the multipart middleware. This would ensure the hardcoded fallback always works, but is less clean since it assumes /tmp is always a valid and accessible path.

  3. Have Omnibus ensure both components inherit the same TMPDIR at the packaging/configuration level, rather than relying on users to configure them independently. This would be a configuration-level fix rather than a code fix.

Option 1 is likely the cleanest approach as it follows the existing architecture where Rails is the source of truth for upload paths.

Related issues and MRs

Edited by 🤖 GitLab Bot 🤖