2020-11-06: merge requests dashboard timing out for some users

Summary

We're seeing more query timeouts for the merge requests dashboard (for instance, https://gitlab.com/dashboard/merge_requests?assignee_username=smcgivern for me) for some users: https://log.gprd.gitlab.net/goto/828433d40a4fc1cd0057724cd2d4694f

The most affected users are at https://log.gprd.gitlab.net/goto/f10f31cfc140642746e8e1d4be29c7aa, but some people can load their dashboard fine.

Context will be added here as we investigate.

Timeline

All times UTC.

2020-11-06

• 10:10 - Sean asks in #is-this-known and Nick creates an issue about handling the timeouts here: gitlab-org/gitlab#277355
• 11:35 - Sean asks in #database due to some odd query plans: https://gitlab.slack.com/archives/C3NBYFJ6N/p1604662538407500
• 13:16 - Sean creates an issue based on that thread: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11840
• 13:39 - smcgivern declares incident in Slack.
• 14:00 - In a call, @bjk-gitlab demonstrates that the query from https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11840 does not time out on any replica.
• 14:07 - @grzesiek creates gitlab-org/gitlab#277386 (closed) linking to the code change that caused this.
• 14:18 - Sean creates a revert MR: gitlab-org/gitlab!47074 (merged)

Click to expand or collapse the Incident Review section.

Incident Review

Summary

1. Service(s) affected:
2. Team attribution:
3. Minutes downtime or degradation:

Metrics

Customer Impact

1. Who was impacted by this incident? (i.e. external customers, internal customers)
2. What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
3. How many customers were affected?
4. If a precise customer impact number is unknown, what is the estimated potential impact?

Incident Response Analysis

1. How was the event detected?
2. How could detection time be improved?
3. How did we reach the point where we knew how to mitigate the impact?
4. How could time to mitigation be improved?

Post Incident Analysis

1. How was the root cause diagnosed?
2. How could time to diagnosis be improved?
3. Do we have an existing backlog item that would've prevented or greatly reduced the impact of this incident?
4. Was this incident triggered by a change (deployment of code or change to infrastructure. If yes, have you linked the issue which represents the change?)?

5 Whys

Lessons Learned

Corrective Actions

Guidelines

• Blameless RCA Guideline

added IncidentActive Source::IMAIncidentDeclare incident severity3 labels

Slack channel here.

assigned to @bjk-gitlab and @brentnewton

changed the description

Possible corrective action: it would also be very helpful if statement timeouts from load balancing hosts included the host somewhere in the exception. Right now we can't tell from Sentry or our logs if this is one host, all of them, or something in between.

We also don't have the actual query in our logs or Sentry. This meant I made a fatal mistake, that thankfully @grzesiek fixed: this was a different query on canary and main. The query I posted was much slower on one replica, but didn't time out, and that was due to not having data in cache. But that was a red herring as far as this issue was concerned.

We actually have it now because of gitlab-org/gitlab#235161 (closed). It didn't show up for this particular exception though 🤔

Thanks, I thought I'd seen something related. I'll look into it more on Monday.

Hi @ops-gitlab-net,

This incident issue does not have any service attribution. Please add one or more of the appropriate service label that are prefixed with Service:.

Please also add a group:: scoped label to help trace to a correct engineering group.

Thanks for your help! 🖤

---

You are welcome to help improve this comment.

added auto updated label

added ServicePostgres label

It seems we had 2 events today correlated with slow queries time spend in queries (mainly for raw controller and merge_requests::content_controller) and 503s:

Grafana

Grafana

Correction - above is not "slow queries" but "time spend in queries".

@grzesiek pointed out gitlab-org/gitlab!45738 (merged).

Here's the timeouts for this on the main stage: https://log.gprd.gitlab.net/goto/1214fb28c57714a41f107c3225b8845e

The production deploy corresponding to this MR started at 09:46 UTC, so that's almost certainly it: https://gitlab.slack.com/archives/C8PKBH3M5/p1604655960184900

@bjk-gitlab ran the query in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11840 on all Patroni nodes and it didn't time out anywhere, so the problem is probably the change from gitlab-org/gitlab!45738 (merged).

I'll create a revert MR.

mentioned in merge request gitlab-org/gitlab!47074 (merged)

added ServiceWeb label and removed ServicePostgres label

changed the description

mentioned in issue on-call-handovers#1111 (closed)

mentioned in issue gitlab-org/release/tasks#1806 (closed)

mentioned in issue on-call-handovers#1112 (closed)

gitlab-org/release/tasks#1806 (comment 443445293) should allow us to mark this as IncidentMitigated, but I wasn't involved in the triage so hopefully someone here can confirm first.

@rspeicher we can not deploy the merge request with the fix because there are active incidents 🤔

mentioned in issue on-call-handovers#1113 (closed)

mentioned in issue on-call-handovers#1114 (closed)

mentioned in issue on-call-handovers#1115 (closed)

mentioned in issue on-call-handovers#1116 (closed)

mentioned in issue on-call-handovers#1117 (closed)

mentioned in issue on-call-handovers#1119 (closed)

Status update: This seems to work now, but we are not sure why.

Leaving the incident active for now till the Revert has been fully deployed (in the process of that, still blocked by some other Q/A related incident: #2997 (closed))

mentioned in issue on-call-handovers#1120 (closed)

mentioned in issue on-call-handovers#1121 (closed)

One of our Merge Requests couldn't be accessed yesterday. This morning, we could access it but the MR shows 0 commit (there should be about 30 commits). Pushing another commit to the branch seemed to have fixed it.

In case anyone have similar problem, just create another commit and push it to the remote branch.

mentioned in issue on-call-handovers#1122 (closed)

I think that gitlab-org/release/tasks#1806 (comment 444389353) explains why it works, but the MR gitlab-org/gitlab!47074 (merged) seems to be stuck on staging. Applying the Pick into auto-deploy label resulted in us cherry-picking this to a deploy branch, thus the merge commit from the merge request might not be deployed, however the revert has been immediately deployed and fixed the problem described in this issue.

I wonder if we should fix out tooling tracking deployments of MRs when Pick into auto-deploy label is being used 🤔 /cc @smcgivern @albertoramos

Yeah, thanks for tracking that down @grzesiek! @albertoramos I think we can close this incident.

mentioned in issue gitlab-org/gitlab#277438 (closed)

added IncidentResolved label and removed IncidentActive label

closed

mentioned in issue reliability-reports#117 (closed)