E2E Validation Strategy for Closed Beta

Artifact Registry — E2E Validation Strategy for Closed Beta

Summary

This issue defines the end-to-end validation strategy, execution plan, and evidence model for Artifact Registry Closed Beta on GitLab.com. Its purpose is to prove that the closed-beta product works as an integrated customer experience, not just as a set of individually completed workstreams. This issue is intended to satisfy the closed-beta readiness gate for end-to-end manual testing, which is explicitly owned by the AR team.

The plan below is intentionally detailed so it can be used by humans and other LLMs to validate whether the strategy is complete, whether the scenario inventory matches closed-beta scope, and whether the resulting evidence is strong enough for a launch recommendation.

Why this matters

Closed Beta already has a fixed scope and an engineering delivery spine. The current program view says scope is fixed, package-side dev complete is targeted for around Aug 14, integration and readiness complete for around Aug 25, and a credible closed beta for the week of Aug 24. That means this issue cannot be a lightweight smoke-test checklist. It needs to verify the real integrated path across product, auth, infra, UI, onboarding, and protocol behavior.

The same roadmap explicitly calls out a real external dependency chain in front of the product surface: hosted formats, the management API, and remote/virtual formats all depend on the real auth/authz path, which in turn depends on token validation, GLAZ, and the Relationships API. A closed-beta E2E strategy therefore has to validate whole flows across boundaries, not only AR-local functionality.

The architecture readiness assessment also calls out that Beta quality assurance needs test coverage evidence and E2E test results.

Goal

Define and execute a closed-beta E2E strategy that answers the following question unambiguously:

Can a real GitLab.com closed-beta customer be onboarded, reach a usable Artifact Registry state, authenticate successfully, and perform the core hosted, remote, and virtual registry workflows for OCI, Maven, and npm across staging and production with sufficient evidence to support launch?

Scope

In scope

The closed-beta product definition includes Maven, npm, and Docker/OCI, plus virtual-registry behavior and the org-level dashboard surface. The engineering roadmap also includes the remote and virtual workstreams for all three formats in the closed-beta build scope: Maven remote/virtual, npm remote/virtual, and Container remote/virtual.

This issue covers:

  • Validation in both Staging and Production.
  • Binaries built, packaged, and deployed to GitLab.com environments.
  • Closed-beta onboarding and enablement flow, including the Organizations dependency, Artifact Registry enablement, registry-handle claim, and the AR UI becoming visible and usable.
  • Successful customer authentication workflows for manual artifact push/pull.
  • Successful hosted push/pull validation for OCI, Maven, and npm.
  • Successful remote-registry setup and validation for OCI, Maven, and npm.
  • Successful virtual-registry setup and validation for OCI, Maven, and npm.
  • Validation of any other capability that is in closed-beta scope per the canonical scope artifacts, rather than what happens to be implemented first.
  • Evidence collection, defect triage, and explicit go/no-go reporting.

Out of scope

The product gate for .com Closed Beta explicitly excludes migration tooling, PyPI/NuGet, dependency firewall, and customer billing from the first closed-beta product scope.

The engineering roadmap also explicitly marks access rules and reductive repository overrides as out of scope for closed beta.

Self-Managed closed-beta execution should be tracked separately. Product milestones define a separate SM closed-beta gate, and the activation/first-run work for .com is not the same as the SM program.

Canonical source of truth

This issue should not become a parallel scope definition. On any conflict, the canonical artifacts win.

What this issue must produce

This issue should produce five concrete outputs:

  • A scenario inventory that fully covers closed-beta scope.
  • A validation matrix that maps each scenario to environment, owner, prerequisite, and evidence.
  • A named execution plan with explicit handoffs across AR, Auth, Orgs, Fulfillment, UI/Monolith, and Infra.
  • A results log for each scenario run.
  • A final launch-readiness summary with pass/fail/blocked status and linked evidence.

Strategy principles

1. Validate customer journeys, not subsystems

The success criterion is not “S08 is done” or “S14 merged.” The success criterion is that a customer can complete a real flow using the system as shipped. This is especially important because the roadmap identifies a long external dependency chain in front of the product surface.

2. Run in both Staging and Production

Staging is the qualification environment. Production is the launch environment. Both must be validated because some failure modes only appear with real GitLab.com routing, infra, auth boundaries, and production configuration.

3. Prove the real auth path

Closed-beta clients do not authenticate with raw GitLab credentials. They exchange a credential for a short-lived JWT, and glab is the user-facing entry point for that exchange in the CC v1 same-boundary model. Any E2E plan that validates only bootstrap/stub auth is insufficient for launch.

4. Cover each format in each delivery mode that is in scope

Hosted validation is not enough. Closed-beta build scope includes remote and virtual format workstreams, so the E2E plan must test hosted, remote, and virtual paths where those are part of the promised product.

5. Require evidence strong enough for audit by another team

Every scenario must leave behind enough evidence that another engineer, EM, PM, or LLM can answer:

  • What was tested?
  • In which environment?
  • Against which build?
  • With which actor and auth mode?
  • What happened?
  • What remains blocked?

Proposed structure

Goal of the issue

Define, execute, and track the end-to-end validation strategy for Artifact Registry Closed Beta on GitLab.com, covering onboarding, authentication, hosted usage, remote usage, virtual usage, and production readiness across staging and production.

Why it matters

Closed Beta requires more than per-workstream completion. The AR roadmap already treats end-to-end manual testing as a readiness gate, and the closed-beta schedule compresses integration and readiness into the period immediately before launch. This issue is the artifact that turns that gate into something explicit, executable, and reviewable.

In scope

  • Staging qualification.
  • Production qualification.
  • GitLab-assisted closed-beta onboarding path.
  • Org visibility and AR UI visibility after enablement.
  • Registry-handle claim and first usable state.
  • Manual authentication flow for closed beta via credential exchange and short-lived JWT.
  • Hosted OCI push/pull.
  • Hosted Maven push/pull.
  • Hosted npm push/pull.
  • Remote OCI setup and consumption.
  • Remote Maven setup and consumption.
  • Remote npm setup and consumption.
  • Virtual OCI setup and consumption.
  • Virtual Maven setup and consumption.
  • Virtual npm setup and consumption.
  • Negative-path validation for auth, authorization, missing setup, and upstream failures.
  • Evidence capture and launch recommendation.

Out of scope

  • GA-only capability validation.
  • Self-Managed closed-beta execution.
  • Features explicitly deferred from closed beta, unless they are needed only as dependencies for an in-scope scenario.

Phases

Phase 0 — Define and staff the matrix

Goal: convert scope into named scenarios with owners, prerequisites, and evidence expectations.

Outputs:

  • Scenario inventory.
  • Owners per lane.
  • Required dependencies list.
  • Environment list.
  • Evidence template.

Phase 1 — Staging qualification

Goal: prove the integrated flow works in staging first.

Exit criteria:

  • All must-pass staging scenarios are green, or have explicitly accepted launch waivers.
  • All blockers are linked and triaged.
  • Production runbook is updated with staging learnings.

Phase 2 — Production qualification

Goal: prove the system as actually deployed on GitLab.com works for the intended closed-beta experience.

Exit criteria:

  • All must-pass production scenarios are green.
  • No Sev1 or launch-blocking defect remains open in the validated paths.
  • Named DRIs have signed off their dependent areas.

Phase 3 — Design-partner validation

Goal: prove the validated flows also work for real customer-like actors, not just internal operators.

Exit criteria:

  • At least one end-to-end success path per supported format.
  • At least one real onboarding/enablement success path.
  • Evidence from actual design-partner or internal-customer rehearsals.
  • Open gaps are documented and either blocked, deferred, or accepted.

Phase 4 — Launch recommendation

Goal: produce the final view for go/no-go.

Outputs:

  • Pass/fail/blocked per scenario.
  • Open-risk summary.
  • Recommendation: Ready / Ready with waivers / Not ready.

Scenario inventory

The scenario IDs below are intentionally stable so they can be referenced in comments, issue updates, spreadsheets, or automated reports.

A. Build, package, deploy

  • E2E-001 — Staging build provenance validated.
  • E2E-002 — Staging package/install/deploy validated.
  • E2E-003 — Production build provenance validated.
  • E2E-004 — Production package/install/deploy validated.
  • E2E-005 — Required dependencies healthy after deploy: DB, object storage, Redis/Valkey, background jobs, auth dependencies, AR UI path.
  • E2E-006 — Roll-forward / rollback confidence for the closed-beta deployment path.

Notes:

  • Infra provisioning, Runway, Fairway, and readiness dependencies are already called out as external workstreams in the roadmap and must be part of the validation preconditions, not treated as invisible background assumptions.

B. Onboarding and first usable state

  • E2E-010 — Starting from a customer with no AR enabled, GitLab-assisted closed-beta onboarding succeeds.
  • E2E-011 — Organization dependency is satisfied and the user can reach the Org landing page.
  • E2E-012 — AR is enabled for the Org.
  • E2E-013 — AR UI is visible in the expected Org navigation path.
  • E2E-014 — Customer claims registry handle successfully.
  • E2E-015 — First usable AR state is reached after handle claim.
  • E2E-016 — Negative path: missing Org / incomplete Org transition blocks correctly and visibly.
  • E2E-017 — Negative path: enablement dependency failure is observable and triaged correctly.

Notes:

  • The closed-beta activation flow is GitLab-assisted, with GitLab handling org creation and AR enablement on the customer’s behalf, followed by Org/AR visibility and registry-handle claim by the customer.

C. Authentication and authorization

  • E2E-020 — Manual token acquisition succeeds via the supported closed-beta client path.
  • E2E-021 — Manual OCI login succeeds.
  • E2E-022 — Manual Maven login/config succeeds.
  • E2E-023 — Manual npm login/config succeeds.
  • E2E-024 — Token expiry / refresh behavior is acceptable for manual workflows.
  • E2E-025 — Authorized actor can perform expected operations.
  • E2E-026 — Unauthorized actor is denied expected operations.
  • E2E-027 — Negative path: wrong credential or wrong scope fails cleanly.
  • E2E-028 — CI-oriented auth path validated where in scope.

Notes:

  • Closed-beta clients exchange a credential for a short-lived JWT; glab is the planned user-facing entry point for this flow in CC v1.
  • The auth/authorization model for closed beta is closed-by-default, with AR-specific roles and additive repository overrides only.

D. Hosted registry workflows

  • E2E-030 — OCI hosted push succeeds.
  • E2E-031 — OCI hosted pull succeeds.
  • E2E-032 — Maven hosted publish succeeds.
  • E2E-033 — Maven hosted consume succeeds.
  • E2E-034 — npm hosted publish succeeds.
  • E2E-035 — npm hosted install succeeds.
  • E2E-036 — Hosted workflow evidence captured through native clients and resulting AR state.
  • E2E-037 — Negative path: denied access fails correctly for each hosted format.

Notes:

  • Hosted npm and hosted OCI are already marked complete in the roadmap, while hosted Maven remains in progress; all three remain part of the closed-beta build scope and should still be revalidated end to end.

E. Remote registry workflows

  • E2E-040 — OCI remote repository can be created/configured and used successfully.
  • E2E-041 — Maven remote repository can be created/configured and used successfully.
  • E2E-042 — npm remote repository can be created/configured and used successfully.
  • E2E-043 — Cache behavior is observable and acceptable for remote flows.
  • E2E-044 — Upstream auth or connectivity failure produces acceptable behavior.
  • E2E-045 — Remote workflows work in staging.
  • E2E-046 — Remote workflows work in production.

Notes:

  • The roadmap explicitly includes Container remote, Maven remote, and npm remote in closed-beta build scope.

F. Virtual registry workflows

  • E2E-050 — OCI virtual repository can be created/configured and used successfully.
  • E2E-051 — Maven virtual repository can be created/configured and used successfully.
  • E2E-052 — npm virtual repository can be created/configured and used successfully.
  • E2E-053 — Virtual resolve and cache behavior match expected closed-beta behavior.
  • E2E-054 — Virtual routing over configured upstreams behaves correctly.
  • E2E-055 — Negative path: broken upstream or missing upstream is handled correctly.

Notes:

  • Product scope explicitly includes virtual-registry behavior in closed beta, and the roadmap includes Container virtual, Maven virtual, and npm virtual workstreams in build scope.

G. UI and management surface

  • E2E-060 — Org navigation shows the AR entry point correctly.
  • E2E-061 — Empty state after activation is correct and actionable.
  • E2E-062 — Repository list / dashboard surface renders correctly for closed-beta scenarios.
  • E2E-063 — Settings / activation surface is correct for the validated phase.
  • E2E-064 — UI reflects success and failure states clearly enough for beta users.

Notes:

  • The activation-flow issue defines the expected entry point, empty-state behavior, and settings accessibility after activation.

H. Readiness, defects, and sign-off

  • E2E-070 — Evidence set assembled for each must-pass scenario.
  • E2E-071 — Blocking defects linked and severity-assessed.
  • E2E-072 — Waivers explicitly recorded, if any.
  • E2E-073 — Final launch recommendation posted.

Validation matrix

The table below is the working skeleton. Owners are intentionally left as To be confirmed unless already unambiguous from the canonical source.

Scenario group Environments Primary owner Supporting owners Status
Build / deploy Staging, Production To be confirmed Infra, AR To be confirmed
Onboarding / enablement Staging, Production To be confirmed Orgs, Fulfillment, Monolith/UI, AR To be confirmed
Auth / authz Staging, Production To be confirmed Auth, AR, CLI To be confirmed
Hosted OCI Staging, Production To be confirmed AR To be confirmed
Hosted Maven Staging, Production To be confirmed AR To be confirmed
Hosted npm Staging, Production To be confirmed AR To be confirmed
Remote OCI Staging, Production To be confirmed AR, Infra To be confirmed
Remote Maven Staging, Production To be confirmed AR, Infra To be confirmed
Remote npm Staging, Production To be confirmed AR, Infra To be confirmed
Virtual OCI Staging, Production To be confirmed AR, Infra To be confirmed
Virtual Maven Staging, Production To be confirmed AR, Infra To be confirmed
Virtual npm Staging, Production To be confirmed AR, Infra To be confirmed
UI / first-run Staging, Production To be confirmed Monolith/UI, UX, AR To be confirmed
Launch summary Production To be confirmed PM, EM, AR, Infra, Auth To be confirmed

Evidence requirements

Each scenario run must produce the following:

  • Date and environment.
  • Git SHA / version / deployment reference.
  • Actor used for the test.
  • Auth method used.
  • Exact scenario ID.
  • Result: Pass / Fail / Blocked.
  • Links to logs, screenshots, CLI transcript, dashboards, or issue comments.
  • If failed or blocked, the linked issue and next action.

A scenario is not complete if the outcome is only described narratively in Slack.

Required dependencies to call out explicitly

The strategy should track these as named dependencies, not hidden assumptions:

  • Organizations onboarding and org visibility path for closed beta.
  • Auth path dependencies: token validation, GLAZ, Relationships API, and client tooling.
  • Infra readiness for CloudSQL, Memorystore, object storage, CDN, and deployment platform.
  • UI and monolith integration for first-run and repository surfaces.
  • Conformance tooling where it gates confidence for format correctness.

Open questions

These should be resolved in comments on this issue rather than guessed in the description:

  • Which scenarios are hard gate vs soft gate for launch?
  • Which production validations must happen with a real design-partner-like actor vs internal test actor?
  • Which auth methods are mandatory for gate closure in this issue vs tracked separately as implementation work?
  • Do we require at least one denied-access authz scenario per format, or one cross-format authz scenario only?
  • What is the minimum required evidence bundle for production sign-off?
  • Which team owns the final execution of cross-service E2E: AR alone, or AR coordinating named DRIs across Auth, Orgs, Infra, and Monolith?
  • Which parts of virtual behavior are required for launch sign-off: resolve only, resolve plus cache, or full topology management?

Acceptance criteria

This issue is complete when all of the following are true:

  • The E2E scenario inventory covers the full .com Closed Beta scope and does not silently omit hosted, remote, virtual, onboarding, auth, or UI validation.
  • Every scenario has an environment, owner, prerequisite set, and evidence location.
  • Staging validation has been run and summarized.
  • Production validation has been run and summarized.
  • At least one successful end-to-end onboarding flow has been validated from “customer does not have AR enabled” to “customer reaches usable AR state.”
  • Manual authentication has been validated using the real closed-beta client path rather than only stub/bootstrap paths.
  • Hosted push/pull succeeds for OCI, Maven, and npm.
  • Remote setup and use succeeds for OCI, Maven, and npm.
  • Virtual setup and use succeeds for OCI, Maven, and npm.
  • Known blockers are either fixed, explicitly waived, or called out as launch blockers.
  • A final recommendation is posted: Ready, Ready with waivers, or Not ready.

Suggested first follow-up tasks under this issue

  • Create the scenario checklist comment with scenario IDs E2E-001 through E2E-073.
  • Assign owners for onboarding, auth, hosted formats, remote formats, virtual formats, and UI validation.
  • Link the current staging and production environments to be used for validation.
  • Link all prerequisite dependency issues and mark which are launch-blocking.
  • Decide the required evidence format for each run.
  • Schedule the first staging qualification pass.
  • Schedule the first production rehearsal.
  • Decide whether weekly demo checkpoints should be part of the execution cadence, since the integrated dev-rig work already highlighted the value of regular cross-team demos for end-to-end progress.

Links

Sources

Edited by Jaime Martinez