ci: add real-backend S3 and GCS driver integration jobs

What

Ports the Container Registry real-backend storage-driver integration jobs to Artifact Registry, adding coverage that previously ran nowhere:

  • test:integration:s3-aws — runs the S3 driver conformance + AC suite against real AWS S3, in a parallel:matrix over S3_AUTH_MODE (key_auth × instance_auth), on the s3-managed_identity_auth runner. The instance_auth leg clears static AWS keys so the SDK uses the VM instance profile.
  • test:integration:gcs-key-creds / test:integration:gcs-instance-creds — run the GCS driver suite against real GCS (no emulator exists; fake-gcs-server is intentionally avoided). key_creds exercises explicit-key V4 URL signing on the default GCP runners; instance_creds runs on the gcs-managed_identity_auth runner and exercises workload identity / the SDK IAM-SignBytes path.

These mirror CR's s3:aws / gcs:key_creds / gcs:instance_creds and run the same conformance + acceptance-criteria suites the existing SeaweedFS/Garage emulator jobs run, but against real cloud backends.

Run rules — the limiting factor is the custom runner

Every job always runs on the default branch (a green main means the real backends passed). On branches/MRs the trigger breadth depends on whether a job is pinned to a scarce custom runner, not on the backend being "real":

Job Runner Branch/MR trigger
s3-seaweedfs, s3-garage (emulators) default wide — broad changes: (driver + testsuites + go.mod/go.sum + CI files), incl. tags
gcs-key-creds (real GCS) default GCP wide — same broad changes: list, incl. tags
s3-aws (real AWS) s3-managed_identity_auth narrow — only when the driver package / testsuites change
gcs-instance-creds (real GCS) gcs-managed_identity_auth narrow — only when the driver package / testsuites change

Why the split:

  • S3 (real AWS) is on a custom runner, so it runs narrow. The AWS runner is needed for both (a) proximity to the bucket + egress-cost optimization and (b) instance-profile / workload-identity auth. Both matrix legs share that one runner, so the whole job stays narrow.
  • GCS key_creds is not on a custom runner, so it runs wide — exactly like the emulator jobs. GitLab's own runners already run on GCP, so bucket proximity and egress cost are non-issues, and key-based auth works on the default runners. There's no custom-runner budget to protect, so it runs on the full changes: set (incl. go.mod / CI files) on MRs, branches, and tags.
  • GCS instance_creds runs narrow because workload-identity auth needs its own managed-identity GCS runner (proximity/cost are already fine on GCP, but instance auth is not available on the default runners).

Custom-runner jobs (s3-aws, gcs-instance-creds) additionally guard against fork pipelines, which can't reach those runners. The default-runner jobs need no such guard — their bucket/credential CI variables are absent in forks, so the suites self-skip there.

Required infrastructure (out of band, not in this MR)

  • Runners: s3-managed_identity_auth, gcs-managed_identity_auth (assumed shared with Container Registry for now — see follow-up below).
  • CI/CD variables (named exactly as the tests env-read them):
    • ARTIFACT_REGISTRY_TEST_S3_BUCKET + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (key_auth leg).
    • ARTIFACT_REGISTRY_TEST_GCS_BUCKET; ARTIFACT_REGISTRY_TEST_GCS_KEY_FILE (File-type, the service-account JSON).
  • Buckets with a ≤24h object-expiry / abort-incomplete-upload lifecycle rule.

Follow-up

These jobs currently reuse Container Registry's shared test buckets. Provisioning dedicated Artifact Registry buckets is tracked in #195 (closed).

Related to #182 (closed)

Edited by Pawel Rozlach

Merge request reports

Loading
Loading