Test Plan for Add-on runner minutes for GitLab.com
Test Plan
Introduction
This test plan is for https://gitlab.com/gitlab-org/gitlab-ee/issues/3314.
Currently, shared CI minutes on GitLab.com are enforced with a hard limit. Once you hit your limit at your particular tier, there's no current path forward. This feature allow users to purchase add-on packs of additional minutes that extend the shared runner limit for the relevant namespace.
Scope
- This feature is for GitLab.com
- The transaction is done on
customers.gitlab.com - Auto-billing is out of scope. Customers will manually purchase packs of minutes.
ACC Matrix
The matrix below identifies the Attributes, Components, and Capabilities relevant to the scope of this test plan.
Attributes (columns) are adverbs or adjectives that describe (at a high level) the qualities testing is meant to ensure Components have.
Components (rows) are nouns that define major parts of the product being tested.
Capabilities link Attributes and Components. They are what your product needs to do to make sure a Component fulfils an Attribute.
This feature includes "CI/CD" and "Subscription" functional area and so they included in the matrix.
For more information see the Google Testing Blog article about the 10 minute test plan and this wiki page from an open-source tool that implements the ACC model.
The numbers indicate the count of Capabilities at each intersection of Attribute and Component
| Secure | Responsive | Intuitive | Reliable | |
|---|---|---|---|---|
| Subscription | 4 | 2 | ||
| CI/CD | 2 |
Capabilities
-
Subscription (Minutes purchase) is
- Intuitive
- It's easy to select from a range of minutes packs to purchase
- It doesn't require unnecessary actions to purchase minutes
- It provides a confirmation message when purchase is successful
- It sends email when all available minutes are used
- Reliable
- It ensures that new purchased minutes are added to the user's minutes.
- It ensures that unused minutes roll over to next month
- Intuitive
-
CI/CD (runners and jobs) is
- Reliable
- Runner is available for use when user's minutes are > 0
- Job is in pending state when available minutes are == 0
- Reliable
Test Plan
Capabilities mentioned above should be used to guide the testing. E2E QA tests would not be needed as the lower level tests would be enough to satisfy the testing needs.