Implement Shared Runner Minute Factors (!27792) · Merge requests · GitLab.org / GitLab

Aleksei Lipniagov requested to merge 210570-change-calc-logic into master Mar 23, 2020

What does this MR do?

After merging !27666 (merged), we have public_projects_minutes_cost_factor and private_projects_minutes_cost_factor fields in the database.

This MR implements the logic changes related to these two fields.

Essentially, here we:

Start counting minutes for public projects too and also include cost factors in the calc: ee/app/services/update_build_minutes_service.rb
Change the way how we pick the builds which meet the quota in the RegisterJobService. Previously, we explicitly allowed all public project, now we keep cost factors in mind for making the decision.
Add a UI piece into admin runner section, only for gitlab.com, which would allow to change the cost factors for a particular runner
Add global ci_minutes_enforce_quota_for_public_projects Feature Flag, responsible picking public project builds without the quota (as it was before) or respect the quota if the cost factor > 0
Add namespace-based Feature Flag ci_minutes_track_for_public_projects to decide if we are updating stats for public projects or not (as it is currently on prod)
Cover the changes with specs

API changes would be considered as a separate issue/MR: #213822

Screenshots

http://localhost:3000/admin/runners/<ID> page (only on gitlab.com):

Does this MR meet the acceptance criteria?

Conformity

Changelog entry
[-] Documentation (if required) - a separate issue: #213096 (closed)
Code review guidelines
Merge request performance guidelines
Style guides
[-] Database guides
Separation of EE specific content

Availability and Testing

To test locally, please mock Gitlab.com? as true, as these changes should affect only .com.
Also, it is recommended to test the namespaces/groups/projects on all levels - public/internal/private.
You will also need to setup a local shared runner.

UI changes may be tested via UI 🙂

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Database query

SELECT
    "ci_builds".*
FROM
    "ci_builds"
    INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id"
    LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id
    LEFT JOIN (
        SELECT
            "ci_builds"."project_id",
            count(*) AS running_builds
        FROM
            "ci_builds"
        WHERE
            "ci_builds"."type" = 'Ci::Build'
            AND ("ci_builds"."status" IN ('running'))
            AND "ci_builds"."runner_id" IN (
                SELECT
                    "ci_runners"."id"
                FROM
                    "ci_runners"
                WHERE
                    "ci_runners"."runner_type" = 1)
            GROUP BY
                "ci_builds"."project_id") AS project_builds ON ci_builds.project_id = project_builds.project_id
WHERE ("ci_builds"."status" IN ('pending'))
AND "ci_builds"."runner_id" IS NULL
AND "projects"."shared_runners_enabled" = TRUE
AND "projects"."pending_delete" = FALSE
AND (project_features.builds_access_level IS NULL
    OR project_features.builds_access_level > 0)
AND "ci_builds"."type" = 'Ci::Build'
AND ("projects"."visibility_level" = 20
    OR (( WITH RECURSIVE "base_and_ancestors" AS (
(
                    SELECT
                        "namespaces".*
                    FROM
                        "namespaces"
                    WHERE (namespaces.id = projects.namespace_id))
            UNION (
                SELECT
                    "namespaces".*
                FROM
                    "namespaces",
                    "base_and_ancestors"
                WHERE
                    "namespaces"."id" = "base_and_ancestors"."parent_id"))
            SELECT
                1
            FROM
                "base_and_ancestors" AS "namespaces"
            LEFT JOIN namespace_statistics ON namespace_statistics.namespace_id = namespaces.id
        WHERE
            "namespaces"."parent_id" IS NULL
            AND (COALESCE(namespaces.shared_runners_minutes_limit, 10, 0) = 0
                OR COALESCE(namespace_statistics.shared_runners_seconds, 0) < COALESCE((namespaces.shared_runners_minutes_limit + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), (10 + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), 0) * 60)) = 1))
ORDER BY
    COALESCE(project_builds.running_builds, 0) ASC,
ci_builds.id ASC

Plan: https://explain.depesz.com/s/CVAw

`legacy_enforce_minutes_for_non_public_projects` query (for comparison)

SELECT
    "ci_builds".*
FROM
    "ci_builds"
    INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id"
    LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id
    LEFT JOIN (
        SELECT
            "ci_builds"."project_id",
            count(*) AS running_builds
        FROM
            "ci_builds"
        WHERE
            "ci_builds"."type" = 'Ci::Build'
            AND ("ci_builds"."status" IN ('running'))
            AND "ci_builds"."runner_id" IN (
                SELECT
                    "ci_runners"."id"
                FROM
                    "ci_runners"
                WHERE
                    "ci_runners"."runner_type" = 1)
            GROUP BY
                "ci_builds"."project_id") AS project_builds ON ci_builds.project_id = project_builds.project_id
WHERE
    "ci_builds"."type" = 'Ci::Build'
    AND ("ci_builds"."status" IN ('pending'))
AND "ci_builds"."runner_id" IS NULL
AND "projects"."shared_runners_enabled" = TRUE
AND "projects"."pending_delete" = FALSE
AND (project_features.builds_access_level IS NULL
    OR project_features.builds_access_level > 0)
AND (projects.visibility_level = 20
    OR ( WITH RECURSIVE "base_and_ancestors" AS (
(
                SELECT
                    "namespaces".*
                FROM
                    "namespaces"
                WHERE (namespaces.id = projects.namespace_id))
        UNION (
            SELECT
                "namespaces".*
            FROM
                "namespaces",
                "base_and_ancestors"
            WHERE
                "namespaces"."id" = "base_and_ancestors"."parent_id"))
        SELECT
            1
        FROM
            "base_and_ancestors" AS "namespaces"
        LEFT JOIN namespace_statistics ON namespace_statistics.namespace_id = namespaces.id
    WHERE
        "namespaces"."parent_id" IS NULL
        AND (COALESCE(namespaces.shared_runners_minutes_limit, 0, 0) = 0
            OR COALESCE(namespace_statistics.shared_runners_seconds, 0) < COALESCE((namespaces.shared_runners_minutes_limit + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), (0 + COALESCE(namespaces.extra_shared_runners_minutes_limit, 0)), 0) * 60)) = 1)
ORDER BY
    COALESCE(project_builds.running_builds, 0) ASC,
ci_builds.id ASC

Plan: https://explain.depesz.com/s/2wvr

Related to #210570 (closed)

Edited May 31, 2022 by 🤖 GitLab Bot 🤖

Implement Shared Runner Minute Factors