2020-03-16 Error occurred when fetching sidebar data

Summary

When viewing MRs or issues that were part of canary, an error message appeared: Error occurred when fetching sidebar data and sidebar fields (e.g. Assignee, Epic) were not loading.

Timeline

All times UTC.

2020-03-16

• 08:16 - cny deployment that included this change: gitlab-org/gitlab!26171 (merged) was started
• 09:48 - users report on Slack the error
• 10:00 - dev-escalation is pulled in
• 10:18 - the MR breaking project query is identified
• 10:31 - chatops command for draining canary is issued
• 10:33 - canary is drained, error goes away

Resources

1. If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)

added incident severity4 labels

Hi @mwasilewski-gitlab,

This incident issue does not have any service attribution. Please add one or more of the appropriate service label that are prefixed with Service:.

Thanks for your help! 🖤

---

You are welcome to help improve this comment.

added auto updated label

assigned to @ntepluhina

created merge request !14 (closed) to address this issue

mentioned in merge request !14 (closed)

I can't reproduce this when logged in. When I'm logged out, I can, and I have this error in response to a GraphQL request:

[{"errors":[{"message":"Type mismatch on variable $iid and argument iid (ID! / String)","locations":[{"line":3,"column":11}],"path":["query","project","issue","iid"],"extensions":{"code":"variableMismatch","variableName":"iid","typeName":"ID!","argumentName":"iid","errorMessage":"Type mismatch"}}]}]

The bad request query looks like:

[{"operationName":null,"variables":{"fullPath":"gitlab-com/gl-infra/production","iid":"1772"},"query":"query ($fullPath: ID!, $iid: ID!) {
  project(fullPath: $fullPath) {
    issue(iid: $iid) {
      iid
      __typename
    }
    __typename\n  }
}
"}]

The good one is:

[{"operationName":null,"variables":{"fullPath":"gitlab-com/gl-infra/production","iid":"1772"},"query":"query ($fullPath: ID!, $iid: ID!) {
  project(fullPath: $fullPath) {
    issue(iid: $iid) {
      iid
      __typename
    }
    __typename
  }
}
"}]

Which looks the same?

There seems to be a recent commit related to this part of the code: gitlab-org/gitlab@bca90be3

The issue is iid in project query for the sidebar has a wrong type of ID! (should be String)

query ($fullPath: ID!, $iid: ID! -> incorrect) {
  project (fullPath: $fullPath) {
    issue (iid: $iid) {
      iid
    }
  }
}

There were some type changes made recently here: gitlab-org/gitlab!26171 (merged)

added severity2 label

removed severity4 label

Bumping this to S2 per: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/doc/development/contributing/issue_workflow.md#severity-labels

@egrieff @smcgivern could we revert this change? iid is used on different GraphQL queries so I assume not only the sidebar could be broken now

unassigned @ntepluhina

@ntepluhina sure, but it's the first I've seen of it 🙂

If the problem is that change, then it's only on canary and we should not promote canary to production.

Ok! I wonder why our GraphQL explorer shows iid type as String (considering the fact the MR changing it to ID was merged) 🤔

The change was only rolled out to canary

It's weird that this is happening for some users and not for others.

I'm on canary but not seeing the problem.

I'm on canary/next visiting gitlab-org/gitlab-workhorse#185 (closed)

It might be cookie related because chrome also logs this:

I think the problem is that if you visit this page, you get canary (because we automatically use canary for gitlab-com). But you don't get canary for /api/graphql automatically. So the difference for me wasn't logged in / logged out, it was the canary cookie.

If you go to https://gitlab.com/-/graphql-explorer and try this:

query ($fullPath: ID!, $iid: ID!) {
  project(fullPath: $fullPath) {
    issue(iid: $iid) {
      iid
      __typename
    }
    __typename
  }
}

With these variables:

{
  "fullPath": "gitlab-com/gl-infra/production",
  "iid": "32028057"
}

It will error if you're not using canary and work if you are using canary.

Also note that the 'IID' there is actually an ID! The IID of this issue is 1772, the ID is 32028057. So this query doesn't return anything even when it doesn't error 🤔

I think you're right @smcgivern

Locally GraphQL explorer shows the correct type for iid

Thanks @ntepluhina!

@egrieff could you create a revert MR and assign to me, please? I think we need to separate the frontend and backend changes here, because it looks like the backend is backwards-compatible (String! still works on canary for iid), but the frontend change needs the backend change to already be everywhere.

We also need to look into why we're passing an ID as an IID because that won't actually work. (gitlab-org/gitlab@bca90be3 (comment 305427583))

So this one is another one of those bugs that only happen with the canary / production code split?

So this one is another one of those bugs that only happen with the canary / production code split?

It could happen with any rolling deploy but canary / production makes it more obvious (and easier to stop), yes 🙂

I added it to the list of recent bugs with this setup in gitlab-com/www-gitlab-com#6556 (comment 303323884). I think we need to do something about this in terms of testing / QA.

I didn't know we forced canary on the GraphQL explorer too.

I didn't know we forced canary on the GraphQL explorer too.

I don't think we do; I was toggling it at https://next.gitlab.com/ to see the different results.

Ah, nevermind. I misunderstood. I thought it was due to GraphQL explorer on Canary and the API request going to non-Canary.

It's actually because this query is only done by Canary HTML/JS pages but it went to non-canary.