Implement Shared Runner Minute Factors
What does this MR do?
After merging !27666 (merged), we have public_projects_minutes_cost_factor
and private_projects_minutes_cost_factor
fields in the database.
This MR implements the logic changes related to these two fields.
Essentially, here we:
- Start counting minutes for public projects too and also include cost factors in the calc:
ee/app/services/update_build_minutes_service.rb
- Change the way how we pick the builds which meet the quota in the
RegisterJobService
. Previously, we explicitly allowed all public project, now we keep cost factors in mind for making the decision. - Add a UI piece into admin runner section, only for
gitlab.com
, which would allow to change the cost factors for a particular runner - Add global
ci_minutes_enforce_quota_for_public_projects
Feature Flag, responsible picking public project builds without the quota (as it was before) or respect the quota if the cost factor > 0 - Add namespace-based Feature Flag
ci_minutes_track_for_public_projects
to decide if we are updating stats for public projects or not (as it is currently on prod) - Cover the changes with specs
API changes would be considered as a separate issue/MR: #213822
Screenshots
http://localhost:3000/admin/runners/<ID>
page (only on gitlab.com
):
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation (if required) - a separate issue: #213096 (closed)
-
Code review guidelines -
Merge request performance guidelines -
Style guides - [-] Database guides
-
Separation of EE specific content
Availability and Testing
To test locally, please mock Gitlab.com?
as true, as these changes should affect only .com
.
Also, it is recommended to test the namespaces/groups/projects on all levels - public/internal/private.
You will also need to setup a local shared runner.
UI changes may be tested via UI
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Database query
SELECT
"ci_builds".*
FROM
"ci_builds"
INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id"
LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id
LEFT JOIN (
SELECT
"ci_builds"."project_id",
count(*) AS running_builds
FROM
"ci_builds"
WHERE
"ci_builds"."type" = 'Ci::Build'
AND ("ci_builds"."status" IN ('running'))
AND "ci_builds"."runner_id" IN (
SELECT
"ci_runners"."id"
FROM
"ci_runners"
WHERE
"ci_runners"."runner_type" = 1)
GROUP BY
"ci_builds"."project_id") AS project_builds ON ci_builds.project_id = project_builds.project_id
WHERE ("ci_builds"."status" IN ('pending'))
AND "ci_builds"."runner_id" IS NULL
AND "projects"."shared_runners_enabled" = TRUE
AND "projects"."pending_delete" = FALSE
AND (project_features.builds_access_level IS NULL
OR project_features.builds_access_level > 0)
AND "ci_builds"."type" = 'Ci::Build'
AND ("projects"."visibility_level" = 20
OR (( WITH RECURSIVE "base_and_ancestors" AS (
(
SELECT
"namespaces".*
FROM
"namespaces"
WHERE (namespaces.id = projects.namespace_id))
UNION (
SELECT
"namespaces".*
FROM
"namespaces",
"base_and_ancestors"
WHERE
"namespaces"."id" = "base_and_ancestors"."parent_id"))
SELECT
1
FROM
"base_and_ancestors" AS "namespaces"
LEFT JOIN namespace_statistics ON namespace_statistics.namespace_id = namespaces.id
WHERE
"namespaces"."parent_id" IS NULL
AND (COALESCE(namespaces.shared_runners_minutes_limit, 10, 0) = 0
OR COALESCE(namespace_statistics.shared_runners_seconds, 0) < COALESCE((namespaces.shared_runners_minutes_limit + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), (10 + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), 0) * 60)) = 1))
ORDER BY
COALESCE(project_builds.running_builds, 0) ASC,
ci_builds.id ASC
Plan: https://explain.depesz.com/s/CVAw
legacy_enforce_minutes_for_non_public_projects
query (for comparison)
SELECT
"ci_builds".*
FROM
"ci_builds"
INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id"
LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id
LEFT JOIN (
SELECT
"ci_builds"."project_id",
count(*) AS running_builds
FROM
"ci_builds"
WHERE
"ci_builds"."type" = 'Ci::Build'
AND ("ci_builds"."status" IN ('running'))
AND "ci_builds"."runner_id" IN (
SELECT
"ci_runners"."id"
FROM
"ci_runners"
WHERE
"ci_runners"."runner_type" = 1)
GROUP BY
"ci_builds"."project_id") AS project_builds ON ci_builds.project_id = project_builds.project_id
WHERE
"ci_builds"."type" = 'Ci::Build'
AND ("ci_builds"."status" IN ('pending'))
AND "ci_builds"."runner_id" IS NULL
AND "projects"."shared_runners_enabled" = TRUE
AND "projects"."pending_delete" = FALSE
AND (project_features.builds_access_level IS NULL
OR project_features.builds_access_level > 0)
AND (projects.visibility_level = 20
OR ( WITH RECURSIVE "base_and_ancestors" AS (
(
SELECT
"namespaces".*
FROM
"namespaces"
WHERE (namespaces.id = projects.namespace_id))
UNION (
SELECT
"namespaces".*
FROM
"namespaces",
"base_and_ancestors"
WHERE
"namespaces"."id" = "base_and_ancestors"."parent_id"))
SELECT
1
FROM
"base_and_ancestors" AS "namespaces"
LEFT JOIN namespace_statistics ON namespace_statistics.namespace_id = namespaces.id
WHERE
"namespaces"."parent_id" IS NULL
AND (COALESCE(namespaces.shared_runners_minutes_limit, 0, 0) = 0
OR COALESCE(namespace_statistics.shared_runners_seconds, 0) < COALESCE((namespaces.shared_runners_minutes_limit + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), (0 + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), 0) * 60)) = 1)
ORDER BY
COALESCE(project_builds.running_builds, 0) ASC,
ci_builds.id ASC
Plan: https://explain.depesz.com/s/2wvr
Related to #210570 (closed)