## Summary On a self-managed instance on a **GCE VM** using GCS object storage with Application Default Credentials (`google_application_default: true`), Sidekiq processes periodically enter a state where **every** object-storage operation fails with: ``` RuntimeError: Your credentials were not found. To set up Application Default Credentials for your environment, see https://cloud.google.com/docs/authentication/external/set-up-adc googleauth (1.14.0) lib/googleauth/application_default.rb:58:in `get_application_default' fog-google (1.29.4) lib/fog/google/shared.rb:177:in `process_application_default_auth' ... ``` The process **never recovers on its own** — only `gitlab-ctl restart sidekiq` fixes it. Meanwhile finishing CI jobs fail with `data_integrity_failure` (which is not auto-retried), traces aren't archived, artifacts can't be read/deleted, and MR exposed-artifacts widgets break. This is a follow-up to #602411 / !252188. That fix (process-level caching of the `Ci::BuildTraceChunks::Fog` connection, unconditional since 19.5 when `cache_ci_build_trace_chunk_fog_connection` was removed) reduces metadata-server traffic but **does not address this failure mode**: after upgrading from 19.4 to 19.5.1 the failures continue, now raised from `cached_connection` itself. ## Environment - GitLab **19.5.1-ee**, Omnibus, single GCE VM (`n4-custom-24-122880`, europe-west1) - Object storage: GCS, consolidated config, `google_application_default: true` (VM service account) - Gems (from 19.5.1 `Gemfile.lock`): `googleauth 1.14.0`, `google-cloud-env 2.2.1`, `fog-google 1.29.4`, `carrierwave 1.3.4` ## Evidence (last 30 days) - ~17,500 `credentials were not found` exceptions in `exceptions_json.log`, in bursts lasting hours (e.g. 2026-10-08 11:00–19:00, 2026-10-22 14:00 → 2026-10-23 13:00, 2026-10-27 07:00 → ongoing). Our runbook remediation is a Sidekiq restart. - Affects **every** code path that builds a Fog GCS connection, not just build trace chunks: | First app frame | Count | |---|---| | `app/models/ci/build_trace_chunks/fog.rb:146` (`SafeRequestStore` path, 19.4) | 9,038 | | `config/initializers/carrierwave_fog_acl_patch.rb:126` (`directory`) — `Ci::DeleteObjectsWorker` | 5,340 | | `config/initializers/carrierwave_fog_acl_patch.rb:111` (`authenticated_url`) | 1,668 | | `app/models/ci/build_trace_chunks/fog.rb:21` (`cached_connection`, **19.5, with the #602411 fix**) | 1,400 | | `lib/object_storage/direct_upload.rb:267` | 32 | - Top callers: `Ci::ArchiveTraceWorker` (9.4k), `Ci::DeleteObjectsWorker` (5.0k), `Ci::ArchiveTracesCronWorker`, `ReactiveCachingWorker`, `POST /api/:version/jobs/request`, `PUT /api/:version/jobs/:id`. - Bursts coincide with a logged `Google::Cloud::Env::MetadataServerNotResponding` (the transient failure that latches `:no`): 2026-10-20 09:52 UTC (burst in the 09:00–10:00 UTC hours) and 2026-10-22 11:53 UTC (burst from the 11:00 UTC hour, ~2,800 errors over the following hours). - The VM's service account is unchanged and a Sidekiq restart restores service immediately, which points at in-process state rather than an IAM/metadata outage. ## Root cause `googleauth` decides whether it is on GCE via `Google::Cloud.env.metadata?` → `ComputeMetadata#check_existence` (`google-cloud-env 2.2.1`, `lib/google/cloud/env/compute_metadata.rb`). The result is memoized **for the life of the process**: ```ruby def check_existence ... current = @existence return current if [:no, :confirmed].include? @existence # :no is terminal ... def post_update_existence success, current_time = nil return if @existence == :confirmed @mutex.synchronize do if success @existence = :confirmed elsif @existence != :confirmed current_time ||= Process.clock_gettime Process::CLOCK_MONOTONIC @existence = :no if current_time > @startup_time + warmup_time # any transient failure after 60s end end end ``` Any `TRANSIENT_EXCEPTIONS` (timeout, `ECONNREFUSED`, …) on a metadata lookup — with the default `open_timeout` of **100 ms**, `request_timeout` 500 ms, 2 retries — after the 60 s warmup window, in a process that hasn't yet had a successful metadata response, sets `@existence = :no` **permanently**. From then on `GCECredentials.on_gce?` returns `false`, `get_application_default` raises `NOT_FOUND_ERROR`, and every `Fog::Storage.new` in that process fails. Nothing in GitLab ever calls `Google::Auth::GCECredentials.reset_cache`. The 100 ms connect timeout is easy to exceed in a busy multi-threaded Ruby process (GVL contention: the thread that issued the connect may not be scheduled in time) or a briefly loaded VM. Newly forked Sidekiq processes (restarts, memory-killer recycles) are especially exposed because they haven't yet reached `:confirmed`. Why !252188 doesn't help: it caches the *connection* after a successful construction, but the failure happens *during* construction (`Fog::Storage.new` → ADC lookup). `Concurrent::Map#compute_if_absent` doesn't cache the exception, so every call retries the ADC lookup — and each retry hits the memoized `:no`. ## Proposed fix Any of these, ideally (1) + (2): 1. **Don't let a transient failure poison the process.** When building a GCS Fog connection with `google_application_default`, rescue the `credentials were not found` error, call `Google::Auth::GCECredentials.reset_cache` (which calls `compute_metadata.reset_existence!`), and retry once. Or reset existence periodically / on error from a central object-storage helper. 2. **Warm up and pin existence at boot** on GCE: call `Google::Cloud.env.compute_metadata.ensure_existence` in an initializer (Puma worker / Sidekiq process start) when GCS ADC is configured, so `@existence` becomes `:confirmed` before any request traffic — once confirmed it can never flip to `:no`. 3. Allow configuring the metadata timeouts (`open_timeout` / `retry_count` / `warmup_time`) for object storage, with saner defaults than 100 ms for a loaded Sidekiq process. 4. Pick up the upstream fix once it lands: this is tracked as https://github.com/googleapis/ruby-cloud-env/issues/110 (open since 2026-08-28, reproduced by a second reporter, not fixed as of `google-cloud-env` 2.4.1). Since there is no upstream fix yet, GitLab-side mitigation (1)/(2) is needed regardless. Separately: `data_integrity_failure` caused by an object-storage *credential* error isn't a data integrity problem and should be retryable. ## Related - #602411, !252188 (process-level Fog connection cache for build trace chunks) - Upstream root cause: https://github.com/googleapis/ruby-cloud-env/issues/110