Improve DB incident management

Database incidents typically have high severity. It is very important to handle those incidents as efficient as possible. Some possible improvements:

• make it easy to escalate and reach out for support/help
  1. better documentation on how to page OnGres (e.g. on the Incident Management handbook)
  2. add paging instructions to #ongres-gitlab slack channel title
  3. ways to reach out to database team on weekends?

corrective action for production#2885 (closed)

added ServicePostgres corrective action workflow-infraTriage + 1 deleted label

marked this issue as related to production#2885 (closed)

mentioned in issue production#2885 (closed)

added DStores-BacklogDBOperations label

mentioned in issue #11749 (moved)

changed the description

We can start with this little improvement @hphilipps : gitlab-com/www-gitlab-com!66968 (merged)

Now if we look for Ongres in the handbook we'll find that entry immediately.

However we should continue to point #2 (closed) up there, that'd be the most important one.

added workflow-infraIn Progress label and removed workflow-infraTriage label

marked this issue as related to #11832 (closed)

This is related to point #2 (closed) from the Description: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11832

removed workflow-infraIn Progress label

added workflow-infraTriage label

added severity3 label

added teamReliability label and removed 1 deleted label

added workflow-infraReady label and removed workflow-infraTriage label

added workflow-infraUnder Review label and removed workflow-infraReady label

This issue still seems valid, although we need to confirm if Ongress is still available for support. Moving it to Ready.

added workflow-infraReady label and removed workflow-infraUnder Review label

added 1 deleted label

Hi @hphilipps,

We have deprecated the use of the ca::refined label in favor of workflow-infra::Ready, so this issue has been relabeled to that effect.

See gitlab-com/www-gitlab-com!100690 (merged) for more info.

removed 1 deleted label

added ReliabilityP4 label

Hi @hphilipps,

We're updating the workflow-infra scoped label to workflow-infraTriage because it meets the following criteria:

• State is OPEN
• It is a corrective action
• It is assigned to teamReliability
• It is workflow-infraReady
• There has been no activity for at least 3 months (since 2022-08-08T11:19:15.242Z)

Perhaps the reason for inactivity is that this corrective action is missing information? Please review and either close the corrective action if it is no longer relevant, or update it and label as workflow-infraReady.

If you have any questions, please reach out to an Infrastructure Manager.

Thanks for your help!

---

You are welcome to help improve this comment.

added workflow-infraTriage label and removed workflow-infraReady label

Closing as I don't think it is relevant anymore.

closed

added workflow-infraDone label and removed workflow-infraTriage label