Skip to content

Fix environment status in merge request widget

Shinya Maeda requested to merge fix-environment-status-in-merge-request-widget into master

What does this MR do?

This MR is to adjust the current implementation between https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22292 and https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22380 in order to accomplish https://gitlab.com/gitlab-org/gitlab-ce/issues/25140 Deliverable

This MR is a follow-up from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22380.

Fix ci_environments_status to get the real latest deployment

When ci_environments_status is accessed, it returns the latest deployment through existing refs (refs/environments/${env_name}/deployments/${deployment_iid}). This means we cannot get running/failed/etc deployments, but only success deployments, because the ref is created when the deployment succeeded.

To get the real latest deployment record (and its status), we have to ask database, instead of gitaly.

Proxy specific deployment statuses

Deployment object has multiple statuses (created/running/success/failed/canceled), however, frontend code can recognize running/success/failed only. Because of that, deployment status in MR widget will be horrible appearance (See https://gitlab.com/gitlab-org/gitlab-ce/issues/53578#further-details) if it's either created or canceled.

Given frontend doesn't allocate capacity in 11.5, we decided to proxy unsupported statuses to supported ones in backend. Specifically, created will be running, canceled will be failed, respectively.

This is a temporary fix and will be followed-up in https://gitlab.com/gitlab-org/gitlab-ce/issues/53578.

Mitigate the error rates of Gitaly DeadlineExceeded (See more https://gitlab.com/gitlab-org/gitlab-ce/issues/51120)

Recently, production engineers/SRE were suffering because the error rates of ci_environments_status endpoint (gitlab-com/gl-infra/production#542 (closed)) was high (about 3000-5000 crash reports/day). The is because ci_environments_status uses git reference to fetch the corresponding deployment record, however, as I explained above, this can be done in DB query. We can expect that we can mitigate the error rates.

Fix users cannot stop their environments

We accidentally removed BuildSuccessWorker invocation in the downstream MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22380. We have to execute BuildSuccessWorker always, otherwise, users cannot stop environments.

Query optimization

Here are the new queries introduced in this fix.

1. environment.deployments.start.find_by_sha(sha)
  SELECT  "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = 34 AND "deployments"."sha" = '0c2ea7d19b24e2a1eb24d9165c0da0fd1cb25595' LIMIT 1;

2. Environment.where(project_id: [mr.source_project_id, mr.target_project_id]).available.with_deployment(mr.diff_head_sha)
  SELECT "environments".* FROM "environments" WHERE "environments"."project_id" IN (15, 15) AND ("environments"."state" IN ('available')) AND (EXISTS (SELECT 1 FROM "deployments" WHERE (deployments.environment_id = environments.id) AND "deployments"."sha" = '0c2ea7d19b24e2a1eb24d9165c0da0fd1cb25595'));

Those queries can be optimized with the following index.

# Index
"index_deployments_on_environment_id_and_sha" btree (environment_id, sha)
# Query plan 1
gitlabhq_development_ce=# explain analyze SELECT  "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = 34 AND "deployments"."sha" = '0c2ea7d19b24e2a1eb24d9165c0da0fd1cb25595' LIMIT 1;
                                                                                 QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.28..8.30 rows=1 width=126) (actual time=0.015..0.015 rows=1 loops=1)
   ->  Index Scan using index_deployments_on_environment_id_and_sha on deployments  (cost=0.28..8.30 rows=1 width=126) (actual time=0.015..0.015 rows=1 loops=1)
         Index Cond: ((environment_id = 34) AND ((sha)::text = '0c2ea7d19b24e2a1eb24d9165c0da0fd1cb25595'::text))
 Planning time: 0.121 ms
 Execution time: 0.034 ms
(5 rows)

# Query plan 2

gitlabhq_development_ce=# explain analyze SELECT "environments".* FROM "environments" WHERE "environments"."project_id" IN (15, 15) AND ("environments"."state" IN ('available')) AND (EXISTS (SELECT 1 FROM "deployments" WHERE (deployments.environment_id = environments.id) AND "deployments"."sha" = '0c2ea7d19b24e2a1eb24d9165c0da0fd1cb25595'));
                                                                                  QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop Semi Join  (cost=0.28..17.74 rows=1 width=105) (actual time=0.035..0.039 rows=1 loops=1)
   ->  Seq Scan on environments  (cost=0.00..1.42 rows=1 width=105) (actual time=0.008..0.009 rows=7 loops=1)
         Filter: ((project_id = ANY ('{15,15}'::integer[])) AND ((state)::text = 'available'::text))
         Rows Removed by Filter: 30
   ->  Index Only Scan using index_deployments_on_environment_id_and_sha on deployments  (cost=0.28..8.30 rows=1 width=4) (actual time=0.004..0.004 rows=0 loops=7)
         Index Cond: ((environment_id = environments.id) AND (sha = '0c2ea7d19b24e2a1eb24d9165c0da0fd1cb25595'::text))
         Heap Fetches: 1
 Planning time: 0.393 ms
 Execution time: 0.056 ms
(9 rows)

What are the relevant issue numbers?

Close https://gitlab.com/gitlab-org/gitlab-ce/issues/25140 Close https://gitlab.com/gitlab-org/gitlab-ce/issues/51120 Related https://gitlab.com/gitlab-org/gitlab-ce/issues/53578

This MR has EE port MR

Does this MR meet the acceptance criteria?

Edited by Shinya Maeda

Merge request reports