Retry GCS default credentials after a latched metadata timeout

What does this MR do and why?

With GCS object storage and google_application_default: true, a single metadata-server timeout makes google-cloud-env cache "not on GCE" for the life of the process. If the process has never had a successful metadata response, every later Fog::Storage.new then fails with Your credentials were not found until the process restarts.

This patches Fog::Google::Shared#process_application_default_auth, which every Fog GCS connection goes through. When it fails with that error and the metadata existence check is latched to :no, the patch calls Google::Auth::GCECredentials.reset_cache and retries once. The reset also restarts the 60 s warmup window, so later failures don't latch again straight away.

If the metadata server timed out less than 0.5 s earlier, the retry gets the cached failure back, because google-cloud-env can't expire a failed lookup during its backoff. The latch is still cleared, so the next call recovers.

The retry is bounded: at most one per call. During a sustained metadata outage, the fresh warmup window means the reset recurs at most about once a minute per process, with metadata requests still limited by google-cloud-env's 0.5 s backoff.

There is no feature flag: the change only runs on a path that currently raises, and the worst case is one extra metadata lookup before the same error.

References

• Related to #636314 (closed)
• Upstream: https://github.com/googleapis/ruby-cloud-env/issues/110

Screenshots or screen recordings

Not applicable, no UI change.

How to set up and validate locally

The failure needs a GCE metadata server. GitLab team members can get a GCP project from the Sandbox Cloud.

I tested on an e2-small Debian 13 VM (Ruby 3.3.8) whose service account can write to a test bucket, with the gem versions from Gemfile.lock. The script waits out the 60 s warmup, blocks the metadata server with iptables to cause a real timeout, unblocks it, then connects and writes and reads an object. Without the patch the process stays latched after the metadata server is back. With it, the next call recovers.

Output

===== PATCH=0
Ruby 3.3.8, patch not loaded
Waiting out the 60s warmup window...
Metadata server blocked
during block: FAILED RuntimeError: Your credentials were not found. To set up Application Default, existence=:no
Metadata server unblocked
after unblock #1: FAILED RuntimeError: Your credentials were not found. To set up Application Default, existence=:no
after unblock #2: FAILED RuntimeError: Your credentials were not found. To set up Application Default, existence=:no
after unblock #3: FAILED RuntimeError: Your credentials were not found. To set up Application Default, existence=:no
===== PATCH=1
Ruby 3.3.8, patch loaded
Waiting out the 60s warmup window...
Metadata server blocked
during block: FAILED RuntimeError: Your credentials were not found. To set up Application Default, existence=:unconfirmed
Metadata server unblocked
after unblock #1: OK (wrote and read back "hello after unblock #1"), existence=:confirmed
after unblock #2: OK (wrote and read back "hello after unblock #2"), existence=:confirmed
after unblock #3: OK (wrote and read back "hello after unblock #3"), existence=:confirmed

Script (latch_test.rb, run as root with PATCH=0|1 BUCKET=... PROJECT=...; patch.rb is the initializer from this MR)

require 'bundler/setup'
require 'googleauth'
require 'fog/google'
require_relative 'patch' if ENV['PATCH'] == '1'

BUCKET = ENV.fetch('BUCKET')
md = Google::Cloud.env.compute_metadata

def connect
  Fog::Storage.new(provider: 'Google', google_project: ENV.fetch('PROJECT'), google_application_default: true)
end

def attempt(label, md)
  storage = connect
  key = "latch-test-#{Process.pid}-#{label}"
  storage.put_object(BUCKET, key, "hello #{label}")
  body = storage.get_object(BUCKET, key)[:body]
  storage.delete_object(BUCKET, key)
  puts "#{label}: OK (wrote and read back #{body.inspect}), existence=#{md.existence_immediate.inspect}"
rescue => e
  puts "#{label}: FAILED #{e.class}: #{e.message.lines.first.strip}, existence=#{md.existence_immediate.inspect}"
end

puts "Ruby #{RUBY_VERSION}, patch #{ENV['PATCH'] == '1' ? 'loaded' : 'not loaded'}"
puts "Waiting out the #{md.warmup_time}s warmup window..."
sleep md.warmup_time + 5

system('iptables -I OUTPUT -d 169.254.169.254 -j DROP', exception: true)
puts 'Metadata server blocked'
attempt('during block', md)

system('iptables -D OUTPUT -d 169.254.169.254 -j DROP', exception: true)
puts 'Metadata server unblocked'
sleep 2 # past google-cloud-env's 0.5s backoff on the failed lookup

3.times { |i| attempt("after unblock ##{i + 1}", md) }

Gemfile pins googleauth 1.14.0, google-cloud-env 2.2.1, fog-google 1.29.4, signet 0.19.0, faraday 2.14.4, faraday-net_http 3.1.0, google-apis-core 0.18.0, google-apis-storage_v1 0.66.0, google-apis-iamcredentials_v1 0.24.0 and jwt 2.10.3.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.