2020-03-16 Error occurred when fetching sidebar data
Summary
When viewing MRs or issues that were part of canary, an error message appeared: Error occurred when fetching sidebar data
and sidebar fields (e.g. Assignee, Epic) were not loading.
Timeline
All times UTC.
2020-03-16
- 08:16 - cny deployment that included this change: gitlab-org/gitlab!26171 (merged) was started
- 09:48 - users report on Slack the error
- 10:00 - dev-escalation is pulled in
- 10:18 - the MR breaking project query is identified
- 10:31 - chatops command for draining canary is issued
- 10:33 - canary is drained, error goes away
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
No timeline items have been added yet.
- Show closed items
Relates to
- gitlab-org/gitlab #326467
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Michal Wasilewski added incident severity4 labels
- Maintainer
This incident issue does not have any service attribution. Please add one or more of the appropriate service label that are prefixed with
Service:
.Thanks for your help!
You are welcome to help improve this comment.
- 🤖 GitLab Bot 🤖 added auto updated label
added auto updated label
- Natalia Tepluhina assigned to @ntepluhina
assigned to @ntepluhina
- Natalia Tepluhina created merge request !14 (closed) to address this issue
created merge request !14 (closed) to address this issue
- Natalia Tepluhina mentioned in merge request !14 (closed)
mentioned in merge request !14 (closed)
- Author Developer
I can't reproduce this when logged in. When I'm logged out, I can, and I have this error in response to a GraphQL request:
[{"errors":[{"message":"Type mismatch on variable $iid and argument iid (ID! / String)","locations":[{"line":3,"column":11}],"path":["query","project","issue","iid"],"extensions":{"code":"variableMismatch","variableName":"iid","typeName":"ID!","argumentName":"iid","errorMessage":"Type mismatch"}}]}]
The bad request query looks like:
[{"operationName":null,"variables":{"fullPath":"gitlab-com/gl-infra/production","iid":"1772"},"query":"query ($fullPath: ID!, $iid: ID!) { project(fullPath: $fullPath) { issue(iid: $iid) { iid __typename } __typename\n } } "}]
The good one is:
[{"operationName":null,"variables":{"fullPath":"gitlab-com/gl-infra/production","iid":"1772"},"query":"query ($fullPath: ID!, $iid: ID!) { project(fullPath: $fullPath) { issue(iid: $iid) { iid __typename } __typename } } "}]
Which looks the same?
Edited by Sean McGivern- Developer
There seems to be a recent commit related to this part of the code: gitlab-org/gitlab@bca90be3
- Developer
The issue is
iid
in project query for the sidebar has a wrong type ofID!
(should beString
)query ($fullPath: ID!, $iid: ID! -> incorrect) { project (fullPath: $fullPath) { issue (iid: $iid) { iid } } }
Edited by Natalia Tepluhina - Developer
There were some type changes made recently here: gitlab-org/gitlab!26171 (merged)
- Brent Newton added severity2 label
added severity2 label
- Brent Newton removed severity4 label
removed severity4 label
- Contributor
- Developer
@egrieff @smcgivern could we revert this change?
iid
is used on different GraphQL queries so I assume not only the sidebar could be broken now - Natalia Tepluhina unassigned @ntepluhina
unassigned @ntepluhina
@ntepluhina sure, but it's the first I've seen of it
If the problem is that change, then it's only on canary and we should not promote canary to production.
Collapse replies - Developer
Ok! I wonder why our GraphQL explorer shows
iid
type asString
(considering the fact the MR changing it toID
was merged)Edited by Natalia Tepluhina
- Author Developer
The change was only rolled out to canary
- Developer
It's weird that this is happening for some users and not for others.
I'm on canary but not seeing the problem.
Collapse replies - Developer
I'm on canary/next visiting gitlab-org/gitlab-workhorse#185 (closed)
I think the problem is that if you visit this page, you get canary (because we automatically use canary for gitlab-com). But you don't get canary for /api/graphql automatically. So the difference for me wasn't logged in / logged out, it was the canary cookie.
If you go to https://gitlab.com/-/graphql-explorer and try this:
query ($fullPath: ID!, $iid: ID!) { project(fullPath: $fullPath) { issue(iid: $iid) { iid __typename } __typename } }
With these variables:
{ "fullPath": "gitlab-com/gl-infra/production", "iid": "32028057" }
It will error if you're not using canary and work if you are using canary.
1Collapse replies Also note that the 'IID' there is actually an ID! The IID of this issue is 1772, the ID is 32028057. So this query doesn't return anything even when it doesn't error
- Developer
I think you're right @smcgivern
Locally GraphQL explorer shows the correct type for
iid
Thanks @ntepluhina!
@egrieff could you create a revert MR and assign to me, please? I think we need to separate the frontend and backend changes here, because it looks like the backend is backwards-compatible (
String!
still works on canary foriid
), but the frontend change needs the backend change to already be everywhere.We also need to look into why we're passing an ID as an IID because that won't actually work. (gitlab-org/gitlab@bca90be3 (comment 305427583))
Edited by Sean McGivern 1- Developer
So this one is another one of those bugs that only happen with the canary / production code split?
So this one is another one of those bugs that only happen with the canary / production code split?
It could happen with any rolling deploy but canary / production makes it more obvious (and easier to stop), yes
Edited by Sean McGivern- Developer
I added it to the list of recent bugs with this setup in gitlab-com/www-gitlab-com#6556 (comment 303323884). I think we need to do something about this in terms of testing / QA.
I didn't know we forced canary on the GraphQL explorer too.
Edited by Heinrich Lee Yu 1 I didn't know we forced canary on the GraphQL explorer too.
I don't think we do; I was toggling it at https://next.gitlab.com/ to see the different results.
- Developer
Ah, nevermind. I misunderstood. I thought it was due to GraphQL explorer on Canary and the API request going to non-Canary.
It's actually because this query is only done by Canary HTML/JS pages but it went to non-canary.
- Author Developer
draining canary at 10:31 UTC
- Author Developer
canary was drained and the erros is no longer showing up so I'll close the incident and we can continue investigating in the RCA issue
- Michal Wasilewski closed
closed
- Sean McGivern mentioned in commit gitlab-org/gitlab@bca90be3
mentioned in commit gitlab-org/gitlab@bca90be3
- Author Developer
incident review issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9516
- Michal Wasilewski mentioned in issue on-call-handovers#425 (closed)
mentioned in issue on-call-handovers#425 (closed)
- Heinrich Lee Yu mentioned in issue gitlab-com/www-gitlab-com#6556 (moved)
mentioned in issue gitlab-com/www-gitlab-com#6556 (moved)
- Michal Wasilewski changed the description
Compare with previous version changed the description
- Eugenia Grieff mentioned in merge request gitlab-org/gitlab!27316 (merged)
mentioned in merge request gitlab-org/gitlab!27316 (merged)
- Developer
MR for reverting gitlab-org/gitlab!26171 (merged) created here -> gitlab-org/gitlab!27316 (merged)
- Sean McGivern mentioned in merge request gitlab-org/gitlab!26171 (merged)
mentioned in merge request gitlab-org/gitlab!26171 (merged)
- Chun Du mentioned in merge request gitlab-com/www-gitlab-com!43618 (closed)
mentioned in merge request gitlab-com/www-gitlab-com!43618 (closed)
- ops-gitlab-net mentioned in issue #1776 (closed)
mentioned in issue #1776 (closed)
- Heinrich Lee Yu mentioned in issue gitlab-org/gitlab#219576 (closed)
mentioned in issue gitlab-org/gitlab#219576 (closed)
- Craig Miskell added ServiceAPI label
added ServiceAPI label
- Maintainer
This incident was closed before the IncidentResolved label was applied applied—or after a review was started by not completed. The issue is being reopened so that it will appear on the Production Incidents board. Please mark it as either IncidentResolved or IncidentReview-Completed before closing.
Please review the Incident Workflow section on the Incident Management handbook page for more information.
/open
- Peter Leitzen mentioned in merge request gitlab-org/gitlab!44347 (merged)
mentioned in merge request gitlab-org/gitlab!44347 (merged)
- Lin Jen-Shin marked this issue as related to gitlab-org/gitlab#326467 (closed)
marked this issue as related to gitlab-org/gitlab#326467 (closed)
New Gitlab user, current Github user here. I wasted an hour trying to follow instructions that said to "click XX on the left sidebar", when there was no such sidebar. Eventually I saw someone comment that it was a font size problem.
Indeed it seems that the width of the left sidebar is being computed incorrectly. It changes in a rather random fashion when I increase the browser's font with CTRL-+, often being too narrow to display the whole icons, and even disappearing completely on occasion.
The best fix, IMHO, would be to remove the sidebar altogether and put its contents in a drop-down menu. Displaying those rarely-used buttons all the time is just a waste or precious screen space.
(I am using Ubuntu 20.04 with the Mate 1.24.0 desktop, on an x86_64 laptop.)
Edited by Jorge StolfiEffects of changing the font size with [CTRL +] and [CTRL -]:
sidebar too narrow: bug-gitlab-sidebar-2.png
sidebar too narow: bug-gitlab-sidebar-1
sidebar missing: bug-gitlab-sidebar-3
Edited by Jorge Stolfi- Owner
@JorgeStolfi This issue is for a (very old) production incident unrelated to UI. I recommend you raise an issue in https://gitlab.com/gitlab-org/gitlab to discuss UI problems.