## Summary Add real-backend driver integration tests for the S06 storage drivers — real AWS S3 (S3 driver) and real GCS (GCS driver) — that currently do not run in CI. - **S3** already has emulator-backed coverage (`test:integration:s3-seaweedfs`, `test:integration:s3-garage`) on every MR; this adds coverage against a real AWS S3 bucket. - **GCS** has **no emulator** (fake-gcs-server is intentionally avoided per the S06 spec), so its integration suite can only run against a real GCS bucket — and today it runs **nowhere** (the GCS port, !552, landed the suite but no CI job). Both suites are `//go:build integration` (`internal/storage/driver/s3/`, `internal/storage/driver/gcs/`) and run the same generic conformance + acceptance-criteria suite parameterized per driver. Real backends cannot run on every MR (cost, shared buckets, managed credentials), so they run on the default branch on a regular schedule (3×/day), mirroring the Container Registry `s3:aws` / `gcs:*` jobs. ## Scope ### S3 — real AWS S3 - Add a `test:integration:s3-aws` job that runs the existing S3 driver integration suite (`go test -tags=integration ./internal/storage/driver/s3/`) against a real AWS S3 bucket. - Run it on the **default branch only**, never on MR pipelines. - Drive it from a **pipeline schedule running 3 times a day** (matching the Container Registry cadence). - Run on the **custom AWS S3 runner** (tag `s3-managed_identity_auth`) — an AWS VM in `us-east-1`, co-located with the test bucket, with managed (instance) credentials configured. - Cover both auth modes in a `parallel:matrix` — `key_auth` (static keys via the AWS default credential chain) and `instance_auth` (VM instance profile / managed identity). The Step 16 (#161) and Step 21 (#166) acceptance criteria already call for this `key_auth` × `instance_auth` matrix on the default branch. ### GCS — real GCS sandbox - Add a `test:integration:gcs` job (matrix legs, e.g. `:key_creds` / `:instance_creds`) that runs the GCS driver integration suite (`go test -tags=integration ./internal/storage/driver/gcs/`) against a real GCS bucket. - **No emulator** — GCS always exercises real GCS. The suite self-skips when `ARTIFACT_REGISTRY_TEST_GCS_BUCKET` is unset, so it is inert wherever the bucket is not configured. - Cover both auth modes in a `parallel:matrix` — `key_creds` × `instance_creds`: - `key_creds` supplies a service-account JSON key (via `ARTIFACT_REGISTRY_TEST_GCS_KEY_FILE`, or `GOOGLE_APPLICATION_CREDENTIALS` for the default chain), exercising explicit-key V4 URL signing. - `instance_creds` uses workload identity (no key file), exercising the SDK's IAM-SignBytes signing path. - Run on the **default branch** on the 3×/day schedule. Per the S06 plan, it may **also** run on MR pipelines when the `changes:` set covers the GCS driver, the testsuites package, the paired Cloud CDN middleware, `go.mod`, `go.sum`, or the CI job file. - Run the `instance_creds` leg on a **managed-identity GCS runner**. ## CI wiring Scheduled pipelines are currently dropped by `workflow:rules` unless they match an explicit pipeline-source anchor (`.if-renovate`, `.if-adr-sync`). This work needs: 1. A new schedule variable + anchor (for example `.if-storage-realbackend-scheduled`) covering the S3-AWS and GCS scheduled jobs, documented alongside the existing schedule anchors in `.gitlab-ci.yml`. 1. A `workflow:rules` entry admitting that scheduled pipeline (before the catch-all `if: $CI_PIPELINE_SOURCE == "schedule" → when: never`). 1. The `test:integration:s3-aws` and `test:integration:gcs` jobs opting into the anchor, plus their default-branch rules (and, for GCS, the `changes:`-filtered MR rule). 1. A 3-times-a-day pipeline schedule in the project CI/CD settings that sets the schedule variable. ## Infrastructure - **S3:** provision the custom AWS S3 runner (AWS VM, `us-east-1`, Docker executor, tag `s3-managed_identity_auth`) with managed credentials for the test bucket. See the Container Registry `docs/custom_runners.md` for the setup procedure. Configure the test bucket with an **`AbortIncompleteMultipartUpload` + object-expiry lifecycle rule** (e.g. delete objects older than 24h) so a failed run cannot accumulate billed storage. - **GCS:** provision a **GCS sandbox bucket** and a **managed-identity GCS runner** (for `instance_creds`). Configure the bucket with an **object-expiry / abort-incomplete-upload lifecycle rule** (e.g. delete objects older than 24h) so failed runs cannot accumulate storage. fake-gcs-server is intentionally **not** used. ## Reference - Container Registry jobs: `.gitlab/ci/integration.yml` — `s3:aws` (`parallel:matrix` `key_auth` × `instance_auth`), `gcs:key_creds`, `gcs:instance_creds`; `rules: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH`; tags for the managed-identity runners. - Container Registry runner setup: `docs/custom_runners.md`. - Existing AR S3 jobs to mirror: `test:integration:s3-seaweedfs` and `test:integration:s3-garage` in `.gitlab-ci.yml`. - GCS suite landed by !552 (S06 Step 17): `internal/storage/driver/gcs/` (`conformance_test.go`, `driver_integration_test.go`); run locally against a real-GCS worker, green, pending this CI wiring. ## Acceptance - **S3:** `test:integration:s3-aws` runs the full S3 driver integration suite green against a real AWS S3 bucket for both `key_auth` and `instance_auth`; runs on default-branch + the 3×/day schedule; does **not** run on MR pipelines; runs on the `s3-managed_identity_auth` runner; the bucket has a lifecycle / auto-cleanup policy. - **GCS:** `test:integration:gcs` runs the full GCS driver integration suite green against a real GCS bucket for both `key_creds` and `instance_creds`; runs on default-branch + the 3×/day schedule (and the `changes:`-filtered MR rule); the `instance_creds` leg runs on a managed-identity GCS runner; the bucket has a lifecycle / auto-cleanup policy. Epic: &22331