ci: add real-backend S3 and GCS driver integration jobs
What
Ports the Container Registry real-backend storage-driver integration jobs to Artifact Registry, adding coverage that previously ran nowhere:
test:integration:s3-aws— runs the S3 driver conformance + AC suite against real AWS S3, in aparallel:matrixoverS3_AUTH_MODE(key_auth×instance_auth), on thes3-managed_identity_authrunner. Theinstance_authleg clears static AWS keys so the SDK uses the VM instance profile.test:integration:gcs-key-creds/test:integration:gcs-instance-creds— run the GCS driver suite against real GCS (no emulator exists; fake-gcs-server is intentionally avoided).key_credsexercises explicit-key V4 URL signing on the default GCP runners;instance_credsruns on thegcs-managed_identity_authrunner and exercises workload identity / the SDK IAM-SignBytes path.
These mirror CR's s3:aws / gcs:key_creds / gcs:instance_creds and run the
same conformance + acceptance-criteria suites the existing SeaweedFS/Garage
emulator jobs run, but against real cloud backends.
Run rules — the limiting factor is the custom runner
Every job always runs on the default branch (a green main means the real
backends passed). On branches/MRs the trigger breadth depends on whether a job
is pinned to a scarce custom runner, not on the backend being "real":
| Job | Runner | Branch/MR trigger |
|---|---|---|
s3-seaweedfs, s3-garage (emulators) |
default | wide — broad changes: (driver + testsuites + go.mod/go.sum + CI files), incl. tags |
gcs-key-creds (real GCS) |
default GCP | wide — same broad changes: list, incl. tags |
s3-aws (real AWS) |
s3-managed_identity_auth |
narrow — only when the driver package / testsuites change |
gcs-instance-creds (real GCS) |
gcs-managed_identity_auth |
narrow — only when the driver package / testsuites change |
Why the split:
- S3 (real AWS) is on a custom runner, so it runs narrow. The AWS runner is needed for both (a) proximity to the bucket + egress-cost optimization and (b) instance-profile / workload-identity auth. Both matrix legs share that one runner, so the whole job stays narrow.
- GCS key_creds is not on a custom runner, so it runs wide — exactly like
the emulator jobs. GitLab's own runners already run on GCP, so bucket
proximity and egress cost are non-issues, and key-based auth works on the
default runners. There's no custom-runner budget to protect, so it runs on the
full
changes:set (incl.go.mod/ CI files) on MRs, branches, and tags. - GCS instance_creds runs narrow because workload-identity auth needs its own managed-identity GCS runner (proximity/cost are already fine on GCP, but instance auth is not available on the default runners).
Custom-runner jobs (s3-aws, gcs-instance-creds) additionally guard against
fork pipelines, which can't reach those runners. The default-runner jobs need no
such guard — their bucket/credential CI variables are absent in forks, so the
suites self-skip there.
Required infrastructure (out of band, not in this MR)
- Runners:
s3-managed_identity_auth,gcs-managed_identity_auth(assumed shared with Container Registry for now — see follow-up below). - CI/CD variables (named exactly as the tests env-read them):
ARTIFACT_REGISTRY_TEST_S3_BUCKET+AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY(key_auth leg).ARTIFACT_REGISTRY_TEST_GCS_BUCKET;ARTIFACT_REGISTRY_TEST_GCS_KEY_FILE(File-type, the service-account JSON).
- Buckets with a ≤24h object-expiry / abort-incomplete-upload lifecycle rule.
Follow-up
These jobs currently reuse Container Registry's shared test buckets. Provisioning dedicated Artifact Registry buckets is tracked in #195 (closed).
Related to #182 (closed)