Skip to content

Add environment auto stop worker

Shinya Maeda requested to merge add-cleanup-worker-for-auto-stopped-environment into master

What does this MR do?

In the previous MR, we've implemented to persist environments.auto_stop_at which indicates when the environment will become stoppable.

This MR implements a cron worker to actually stop the environments with the value.

You can also find PoC !18115 (closed). It's already been reviewed by PM and UX.

Related: #20956 (closed)

Query for searching stop actions in batch (environments.stop_actions)

WITH "deployments_with_stop_action" AS (
    SELECT
        DISTINCT ON (environment_id) deployments.*
    FROM
        "deployments"
    WHERE
        "deployments"."environment_id" IN (
            SELECT
                "environments"."id"
            FROM
                "environments"
            WHERE
                "environments"."environment_type" IS NOT NULL
                AND ("environments"."state" IN ('available'))
            LIMIT
                100
        )
        AND "deployments"."on_stop" IS NOT NULL
        AND "deployments"."deployable_id" IS NOT NULL
        AND "deployments"."status" = 2
    ORDER BY
        environment_id,
        deployments.id DESC
)
SELECT
    "ci_builds".*
FROM
    "ci_builds"
    INNER JOIN "deployments_with_stop_action" ON "ci_builds"."project_id" = "deployments_with_stop_action"."project_id"
    AND "ci_builds"."ref" = "deployments_with_stop_action"."ref"
    AND "ci_builds"."name" = "deployments_with_stop_action"."on_stop"
WHERE
    "ci_builds"."type" = 'Ci::Build'
    AND "ci_builds"."commit_id" IN (
        SELECT
            commit_id
        FROM
            "ci_builds"
            INNER JOIN "deployments_with_stop_action" ON "ci_builds"."id" = "deployments_with_stop_action"."deployable_id"
    )
    AND "ci_builds"."status" IN ('manual', 'scheduled')

Query plan

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------
 Hash Join  (cost=24981.25..285835.54 rows=1 width=1544) (actual time=13.944..18.644 rows=69 loops=1)
   Hash Cond: ((ci_builds.project_id = deployments_with_stop_action.project_id) AND ((ci_builds.ref)::text = (deployments_with_stop_action.ref)::text) AND ((ci_builds.name)::text = (deployments_with_stop_action.
on_stop)::text))
   CTE deployments_with_stop_action
     ->  Unique  (cost=20329.79..20336.13 rows=1269 width=138) (actual time=9.767..9.808 rows=72 loops=1)
           ->  Sort  (cost=20329.79..20332.96 rows=1269 width=138) (actual time=9.767..9.777 rows=150 loops=1)
                 Sort Key: deployments.environment_id, deployments.id DESC
                 Sort Method: quicksort  Memory: 64kB
                 ->  Nested Loop  (cost=15.42..20264.38 rows=1269 width=138) (actual time=2.641..9.609 rows=150 loops=1)
                       ->  HashAggregate  (cost=14.85..15.85 rows=100 width=4) (actual time=2.562..2.585 rows=100 loops=1)
                             Group Key: environments.id
                             ->  Limit  (cost=0.43..13.60 rows=100 width=4) (actual time=0.078..2.529 rows=100 loops=1)
                                   ->  Index Scan using index_environments_on_project_id_state_environment_type on environments  (cost=0.43..59279.33 rows=449974 width=4) (actual time=0.078..2.518 rows=100 loops
=1)
                                         Index Cond: (((state)::text = 'available'::text) AND (environment_type IS NOT NULL))
                       ->  Index Scan using index_deployments_on_environment_id_and_status on deployments  (cost=0.56..202.36 rows=13 width=138) (actual time=0.053..0.070 rows=2 loops=100)
                             Index Cond: ((environment_id = environments.id) AND (status = 2))
                             Filter: ((on_stop IS NOT NULL) AND (deployable_id IS NOT NULL))
                             Rows Removed by Filter: 1
   ->  Nested Loop  (cost=4597.52..17695.86 rows=512157 width=1544) (actual time=4.027..8.638 rows=141 loops=1)
         ->  HashAggregate  (cost=4596.95..4609.64 rows=1269 width=4) (actual time=3.916..3.943 rows=72 loops=1)
               Group Key: ci_builds_1.commit_id
               ->  Nested Loop  (cost=0.57..4593.78 rows=1269 width=4) (actual time=0.182..3.877 rows=72 loops=1)
                     ->  CTE Scan on deployments_with_stop_action deployments_with_stop_action_1  (cost=0.00..25.38 rows=1269 width=4) (actual time=0.001..0.020 rows=72 loops=1)
                     ->  Index Scan using ci_builds_pkey on ci_builds ci_builds_1  (cost=0.57..3.59 rows=1 width=8) (actual time=0.050..0.053 rows=1 loops=72)
                           Index Cond: (id = deployments_with_stop_action_1.deployable_id)
         ->  Index Scan using index_ci_builds_on_commit_id_and_status_and_type on ci_builds  (cost=0.57..10.20 rows=11 width=1544) (actual time=0.045..0.063 rows=2 loops=72)
               Index Cond: ((commit_id = ci_builds_1.commit_id) AND ((status)::text = ANY ('{manual,scheduled}'::text[])) AND ((type)::text = 'Ci::Build'::text))
   ->  Hash  (cost=25.38..25.38 rows=1269 width=68) (actual time=9.875..9.875 rows=72 loops=1)
         Buckets: 2048  Batches: 1  Memory Usage: 22kB
         ->  CTE Scan on deployments_with_stop_action  (cost=0.00..25.38 rows=1269 width=68) (actual time=9.770..9.842 rows=72 loops=1)
 Planning time: 19.017 ms
 Execution time: 18.883 ms
(31 rows)

Feature Flag

This feature is built behind auto_stop_environments feature flag and disabled by default until we make sure it's deemed stable.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Craig Norris

Merge request reports