Investigate Package incidents to strategically increase coverage of end-to-end tests

1. Investigate package-related incidents and gather evidence on areas of code that need more coverage, especially at the end-to-end level
2. Create issues to track the test scenarios
3. Prioritise these tests for the upcoming milestones

Incidents 🔥

by historical order

⚠ corrective actions listed here are focused on testing and not including reverts/fixes

2021-02-18: NPM builds broken on gitlab.com severity2

• Infra issue: gitlab-com/gl-infra/production#3661 (closed)

• A retrospective: #33685 (comment 513110803)

• Corrective actions:

  1. Corrective actions resulting from S2 incident - NPM builds not working
  2. Expand the test conditions for all package managers
  3. Add a guideline for bugs severity analysis on the Package Group team page

2021-01-12: Degraded latency on registry server SLI severity4

• Infra issue: gitlab-com/gl-infra/production#3312 (closed)
• Explanatory thread 🧵

2021-02-05: Dependency proxy unable to pull images severity2

• Infra issue: gitlab-com/gl-infra/production#3521 (closed)

• Corrective actions:

  1. Add section about testing uploads to package team
  2. Explore object-storage QA opportunities for package stage features

2020-10-24 Cloud storage slow causing several apdex/error issues severity2

• Infra issue: gitlab-com/gl-infra/production#2887 (closed)

2020-10-15: Unable to upload Maven packages due to Rack (ruby gem) upgrade severity2

• Infra issue: gitlab-com/gl-infra/production#2829 (closed)

• Corrective actions:

  1. Add Gradle coverage for deploying a Maven package

2020-08-24: Maven package size limit breaking pipelines severity1

• Infra issue: gitlab-com/gl-infra/production#2563 (closed)

• Follow up items

• Corrective actions:

  1. Add testing guidelines for package features

What do we have at the end-to-end level

.. in relation to the above ☝ incidents

Test	Spec	Coverage	Improvements
Package NPM registry publishes an npm package and then deletes it	qa/specs/features/browser_ui/5_package/npm_registry_spec.rb	uses a docker container to push the npm package to the package registry. Tests delete functionality via the UI. Uses only the instance-level endpoint. Package size is	use bigger packages by importing real projects/packages to upload; test endpoint at project level; test pull/install of NPM package; test using yarn; use CI
Package Maven Repository with Gradle	/qa/specs/features/browser_ui/5_package/maven_gradle_repository_spec.rb	publishes a package using Gradle and using CI/CD. Tests delete functionality via the UI. Uses the project level endpoint. Package size is	use Gradle to install a package;
Package Maven Repository publishes a maven package and deletes it	/qa/specs/features/browser_ui/5_package/maven_repository_spec.rb	publishes a Maven package using a docker container. Tests delete functionality via the UI. Uses the project-level endpoint. Package size is	use bigger packages by importing real projects/packages to upload; test endpoint at project level; test pull/install of Maven package; test endpoint at instance level; test publishing a package with the same name and version;
Package Maven Repository publishes and downloads a maven package via CI	/qa/specs/features/browser_ui/5_package/maven_repository_spec.rb	publishes a Maven package using CI/CD. Tests delete functionality via the UI. Uses project level endpoint. Package size is	🔺
Package Container Registry pushes project image to the container registry and deletes tag	/qa/specs/features/browser_ui/5_package/container_registry_spec.rb	scoped to run only in staging (and to be promoted to other environments). Uses CI/CD. Tests delete tag functionality via the UI. Size pushed	TBD

Aditional coverage

• PyPI Repository
• NuGet Repository
• Composer Repository
• Conan Repository
• Generic Repository

Not covered - Dependency Proxy 🚥

Scheduled efforts:

• QA tests for the dependency proxy