CI Minutes Cost Factors: QA Track
To finish the development of the CI Minutes Cost Factors, we need to implement the QA Track of this feature.
The doc on the Cost Factors: gitlab-com/runbooks!2293 (merged)
The main implementation issue: #210570 (closed)
If I understand correctly, the final outcome of QA Track
should be a set of automated Feature tests.
We will appreciate the help from the QA team here.
I tried to describe how I see possible End-to-End scenarios:
public_cost_factor=0.0, private_cost_factor=0.0
Scenario 1. Setup:
-
Gitlab.com?
is stubbed to returntrue
- A user, their projects: public, internal and private
- Every project has a pipeline configured (a short one
~ 5s
is preferred) - Cost factors:
public=0.0, private=0.0
, either set via the code, or via UI by the instance Admin following the doc: gitlab-com/runbooks!2293 (merged)
Scenario:
- They visit
Quotas
page, check that total minutes spent is0
(or some other fixed value) - They run pipelines in every project
- They check that total minutes didn't change on the
Quotas
page
Scenario 2. public_cost_factor=0.0, private_cost_factor=1.0
(Gitlab.com now)
public_cost_factor=0.0, private_cost_factor=1.0
(Gitlab.com now)Setup: same, except the different cost factors
Scenario:
They visitQuotas
page, check that total minutes spent is0
(or some other fixed value)They run pipelines in publicThey check that total minutes didn't change on theQuotas
pageThey run pipelines in privateThey check that total minutes increased on theQuotas
page (by the approximate total job length - this is tricky, as it may vary, maybe to check that the increment is in some "normal" interval"They run pipelines in internalCheck that total minutes also increased, as with private
public_cost_factor=1.0, private_cost_factor=1.0
(Gitlab.com in the future)
Scenario 3. Setup: same, except the different cost factors
Scenario:
- They visit
Quotas
page, check that total minutes spent is0
(or some other fixed value) - They run the pipeline in each project and check the quota increment after each run
Scenario 4. public=100.0?, private=0.01
(Trying something different from 1.0
and 0.0
)
public=100.0?, private=0.01
(Trying something different from 1.0
and 0.0
)Similar idea, but to check that different factors work. Probably, it would be easier to do that with some extreme values where we should check that the increment was noticeable. (e.g. > 10 minutes
for a short run or something like this).
Scenario 5 (not .com
)
.com
)Setup: without mocking .com
. Factors set to 0.0/1.0
(defaults).
Scenario: run all pipelines, check no quota changes.
This Cost Factor logic is only supported by .com
, we may want to check it in Feature spec.