Add Missed SLA label via Cron job

What does this MR do?

This adds a cron job and related services to add a missed::SLA label to the incidents which have missed their given SLA times.

To do this we've added a few services:

• IncidentManagement::CreateIncidentSlaExceededLabelService - creates the missed::SLA label in the project
• IncidentManagement::ApplyIncidentSlaExceededLabelService - applies the label to the incidents

This is done via a Cron worker:

• IncidentManagement::IncidentSlaExceededCheckWorker

To aid this we've added a new database scope to IndcidentSla

• IncidentSla.exceeded - This finds incident SLAs where the due_at time has expired, and the corresponding Issue is open.

Query for IncidentSla.exceeded scope:

SELECT "incident_slas".* FROM "incident_slas" INNER JOIN "issues" ON "issues"."id" = "incident_slas"."issue_id" WHERE ("issues"."state_id" IN (1)) AND (due_at < '2020-10-08 05:06:27.511481')

Explain: https://explain.depesz.com/s/SNPr

EXPLAIN for: SELECT "incident_slas".* FROM "incident_slas" INNER JOIN "issues" ON "issues"."id" = "incident_slas"."issue_id" WHERE ("issues"."state_id" IN (1)) AND (due_at < '2020-10-08 05:06:27.511481')
                                                                  QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
 Hash Join  (cost=23.53..58.28 rows=272 width=16)
   Hash Cond: (incident_slas.issue_id = issues.id)
   ->  Seq Scan on incident_slas  (cost=0.00..33.12 rows=617 width=16)
         Filter: (due_at < '2020-10-08 05:06:27.511481+00'::timestamp with time zone)
   ->  Hash  (cost=20.91..20.91 rows=210 width=4)
         ->  Index Only Scan using idx_issues_on_project_id_and_updated_at_and_id_and_state_id on issues  (cost=0.27..20.91 rows=210 width=4)
               Index Cond: (state_id = 1)
(7 rows)

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

Related to #241663 (closed)

changed milestone to %13.5

added Category:Incident Management Deliverable GitLab Premium backend devopsmonitor direction documentation frontend grouprespond release post item release post itemin review sectionops typefeature workflowin dev labels

	2 Warnings
	This merge request includes more than 10 commits. Each commit should meet the following criteria: Have a well-written commit message. Has all tests passing when used on its own (e.g. when using git checkout SHA). Can be reverted on its own without also requiring the revert of commit that came before it. Is small enough that it can be reviewed in isolation in under 30 minutes or so. If this merge request contains commits that do not meet this criteria and/or contains intermediate work, please rebase these commits into a smaller number of commits or split this merge request into multiple smaller merge requests.
	This MR has a Changelog file outside ee/, but code changes in ee/. Consider moving the Changelog file into ee/.

	1 Message
	This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

Documentation review

The following files require a review from a technical writer:

• doc/operations/incident_management/incidents.md

The review does not need to block merging this merge request. See the:

• Technical Writers assignments for the appropriate technical writer for this review.
• Documentation workflows for information on when to assign a merge request for review.

Reviewer roulette

Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited, or the chosen person is unavailable.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.

Category	Reviewer	Maintainer
backend	Mario de la Ossa (@mdelaossa) (UTC-6, 19 hours behind @seanarnold)	Peter Leitzen (@splattael) (UTC+2, 11 hours behind @seanarnold)
database	Marius Bobin (@mbobin) (UTC+3, 10 hours behind @seanarnold)	Adam Hegyi (@ahegyi) (UTC+2, 11 hours behind @seanarnold)

Sidekiq queue changes

This merge request contains changes to Sidekiq queues. Please follow the documentation on changing a queue's urgency.

These queues were added:

• cronjob:incident_management_incident_sla_exceeded_check
• incident_management_apply_incident_sla_exceeded_label

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 4 commits

• b5d49120...8d424da0 - 2 commits from branch 241663-incident-sla-logic
• a3870ab4 - Add exceeded scope to Incident SLA
• 338a2142 - Add SLA check cron job

Compare with previous version

added 556 commits

• 338a2142...0e2e5a45 - 554 commits from branch 241663-incident-sla-logic
• 3f40d369 - Add exceeded scope to Incident SLA
• ae906e0d - Add SLA check cron job

Compare with previous version

added 4 commits

• e1fe31fb - 1 commit from branch 241663-incident-sla-logic
• 214514af - Add exceeded scope to Incident SLA
• 83adcfae - Add SLA check cron job
• 41e7a6e2 - Add label system note (resource event)

Compare with previous version

unmarked as a Work In Progress

changed title from Draft: Resolve "Service level Agreement countdown timer for Incidents" to Add Missed SLA label via Cron job

changed the description

added 7 commits

• 41e7a6e2...00c031a7 - 3 commits from branch 241663-incident-sla-logic
• ed3c4e4d - Add exceeded scope to Incident SLA
• c6e698d1 - Add SLA check cron job
• 1240d165 - Add label system note (resource event)
• 189c6073 - Fix spec file name

Compare with previous version

@syasonik Are you able to do the backend review here?

Note this is waiting for 2 MRs in front of it.. so you could treat this as a light pass if you want.

added database label

added databasereview pending label

added 7 commits

• 189c6073...0da19084 - 3 commits from branch 241663-incident-sla-logic
• f1a12c56 - Add exceeded scope to Incident SLA
• d770f849 - Add SLA check cron job
• ab8a9733 - Add label system note (resource event)
• 9c399d53 - Fix spec file name

Compare with previous version

added 27 commits

• 9c399d53...370527d2 - 23 commits from branch 241663-incident-sla-logic
• c7eb1d11 - Add exceeded scope to Incident SLA
• 7467f858 - Add SLA check cron job
• dbde67b1 - Add label system note (resource event)
• b943f4f7 - Fix spec file name

Compare with previous version

added 1 commit

• cc37b665 - Add tweaks to worker, service, specs

Compare with previous version

added 1 commit

• 04e2ec34 - Use path to create URL

Compare with previous version

added 453 commits

• 04e2ec34...6b0b4843 - 446 commits from branch 241663-incident-sla-logic
• 91440c14 - Add EE changes to controller for SLA
• ddbcc140 - Add exceeded scope to Incident SLA
• 3e3f8e64 - Add SLA check cron job
• a4c20ae1 - Add label system note (resource event)
• 836fb63a - Fix spec file name
• 35cdcb13 - Add tweaks to worker, service, specs
• 62c4629f - Use path to create URL

Compare with previous version

assigned to @syasonik

@mbobin Are you able to database review the new scope please?

assigned to @mbobin

unassigned @mbobin

unassigned @syasonik

added 12 commits

• 62c4629f...0141da94 - 5 commits from branch 241663-incident-sla-logic
• 32384f40 - Add EE changes to controller for SLA
• 61864aaa - Add exceeded scope to Incident SLA
• 4c462816 - Add SLA check cron job
• 260baa79 - Add label system note (resource event)
• efedd4ea - Fix spec file name
• c3f68e8c - Add tweaks to worker, service, specs
• 97195678 - Use path to create URL

Compare with previous version

added 10 commits

• 70f0bfea - 1 commit from branch 241663-incident-sla-logic
• bdbacad5 - Fix import/export models
• b7dd0826 - Fix typo
• 9d7f1f7b - Add EE changes to controller for SLA
• 587d609e - Add exceeded scope to Incident SLA
• 39085ac5 - Add SLA check cron job
• df5cbbc0 - Add label system note (resource event)
• 2bb624bc - Fix spec file name
• a621b6fb - Add tweaks to worker, service, specs
• 7e9578db - Use path to create URL

Compare with previous version

added 1 commit

• 30ddf27d - Tweak some spec syntax

Compare with previous version

added 11 commits

• 1fa42416 - 1 commit from branch 241663-incident-sla-logic
• ddd402ad - Fix import/export models
• 589a2970 - Fix typo
• 3ea007cb - Add EE changes to controller for SLA
• 8ec88463 - Add exceeded scope to Incident SLA
• 1848d1f5 - Add SLA check cron job
• 3bb40519 - Add label system note (resource event)
• aa4d7025 - Fix spec file name
• 932a0523 - Add tweaks to worker, service, specs
• 9c17ad7e - Use path to create URL
• 52676608 - Tweak some spec syntax

Compare with previous version

added 11 commits

• 441a71a3 - 1 commit from branch 241663-incident-sla-logic
• 1a6af185 - Fix import/export models
• 9c93deaf - Fix typo
• 4f4f6200 - Add EE changes to controller for SLA
• 3c21c819 - Add exceeded scope to Incident SLA
• fe4c6105 - Add SLA check cron job
• f23f4c0b - Add label system note (resource event)
• 6bf69b8e - Fix spec file name
• 4b183047 - Add tweaks to worker, service, specs
• 6396cf47 - Use path to create URL
• edbaa551 - Tweak some spec syntax

Compare with previous version

added 1 commit

• 72f53ae7 - Move apply incident label into worker

Compare with previous version

added 1 commit

• 8897d4a9 - Remove spec file

Compare with previous version

added 1 commit

• eb5fc64a - Fix issuable sla scope

Compare with previous version

added 1 commit

• 173d16e3 - Fix spec failures

Compare with previous version

assigned to @syasonik

assigned to @mbobin and unassigned @syasonik

added 2 commits

• 6e43b28f - Move apply incident label into worker
• 0216ac22 - Fix issuable sla scope

Compare with previous version

added 269 commits

• 0216ac22...4b0dc2d6 - 260 commits from branch 241663-incident-sla-logic
• 3ddb2222 - Add exceeded scope to Incident SLA
• 7c52d36f - Add SLA check cron job
• 4a2c9574 - Add label system note (resource event)
• c36f7063 - Fix spec file name
• d64ec076 - Add tweaks to worker, service, specs
• 275cf37b - Use path to create URL
• 1cced1c9 - Tweak some spec syntax
• 3dc453e6 - Move apply incident label into worker
• 968754bb - Fix issuable sla scope

Compare with previous version

assigned to @syasonik

added databasereviewed label and removed databasereview pending label

unassigned @mbobin

unassigned @syasonik

assigned to @aqualls

assigned to @syasonik

added 1 commit

• e0cc075f - Update specs, add changelog

Compare with previous version

added 1 commit

• df9b91a7 - 1 commit from branch 241663-incident-sla-logic

Compare with previous version

changed target branch from 241663-incident-sla-logic to master

approved this merge request

unassigned @syasonik

added 1 commit

• 821212e0 - Use find_each instead of find_in_batches

Compare with previous version

added 216 commits

• 821212e0...4d8b02a4 - 204 commits from branch master
• 99e12104 - Add exceeded scope to Incident SLA
• fd262a4f - Add SLA check cron job
• 152dcfbd - Add label system note (resource event)
• 431fc543 - Fix spec file name
• 62067564 - Add tweaks to worker, service, specs
• 2d77fed6 - Use path to create URL
• c82d8413 - Tweak some spec syntax
• e2c3cc8f - Move apply incident label into worker
• f983eb8c - Fix issuable sla scope
• b0628d95 - Update specs, add changelog
• a1af0d9d - Tweak specs and remove label return
• 5dfe4b69 - Use find_each instead of find_in_batches

Compare with previous version

Generated by Danger

@cablett Are you able to maintainer backend review this please?

@tigerwnz Are you able to maintainer database review this please?

assigned to @cablett and @tigerwnz

added 1 commit

• 368169b7 - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version

approved this merge request

unassigned @aqualls

added Technical Writing docsfeature twfinished labels

approved this merge request

added databaseapproved label and removed databasereviewed label

unassigned @tigerwnz

added 4 commits

• fd2125dc - Update specs
• 0dd0f626 - Tweak specs and remove label return
• 4be60bab - Use find_each instead of find_in_batches
• 654532de - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version

Great work @seanarnold! I left some very minor nits, but nothing blocking.

unassigned @cablett

added 5 commits

• 084cd09f - Update sla label specs
• 36d910a8 - Tweak specs and remove label return
• 1f6acbed - Use find_each instead of find_in_batches
• 59b3d364 - Apply 1 suggestion(s) to 1 file(s)
• ebbae5d3 - Tweak spec and changelog title

Compare with previous version

resolved all threads

marked the checklist item Changelog entry as completed

marked the checklist item Documentation (if required) as completed

marked the checklist item Code review guidelines as completed

marked the checklist item Style guides as completed

marked the checklist item Separation of EE specific content as completed

marked the checklist item Database guides as completed

marked the checklist item Merge request performance guidelines as completed

approved this merge request

Fantastic work @seanarnold !

Thank you for reviewing @mbobin @tigerwnz @syasonik!

enabled an automatic merge when the pipeline for be721a97 succeeds

merged

added workflowstaging label and removed workflowin dev label

added workflowcanary label and removed workflowstaging label

added workflowproduction label and removed workflowcanary label

mentioned in issue #241663 (closed)

This merge request has been deployed to the pre.gitlab.com environment, and will be included in the upcoming self-managed GitLab 13.5.0 release.

---

This comment is generated automatically using the Release Tools project.

added published label

This merge request has been deployed to the release.gitlab.net environment, and will be included in the upcoming self-managed GitLab 13.5.0 release.

---

This comment is generated automatically using the Release Tools project.