2023-04-13: QA pipelines failing on gstg and gstg-cny

added an incident timeline event

added IncidentActive Source::IMAIncidentDeclare a:GitalyServiceGoserverApdexSLOViolationSingleNode blocks deployments blocks feature-flags incident severity2 labels

assigned to @nmccorrison

changed the description

changed the severity to High - S2

added a resource link

The failed test is from a missing "Remove member" button on the group management page. Investigating if this is from a recently enabled feature flag

The failed tests show the the user doesn’t have owner/maintainer access, and therefore doesn’t have the ... button that provides access to the remove member button. Has the role of the E2E user changed?

The failure in browser_ui/9_data_stores/user/parent_group_access_termination_spec.rb appears to be due to the user not having a menu dropdown option to 'remove' a user from the group.

I was temporarily able to reproduce this locally against staging, but it has since begun to work as expected- gitlab-org/gitlab#407224 (comment 1351948731)

However, we're seeing this fail in staging/canary pipelines since.

Noting also that it did pass once - https://ops.gitlab.net/gitlab-org/quality/staging/-/pipelines/1862161 here, but has failed in builds both prior and after this.

There's also a second test failing - gitlab-org/gitlab#407214 (closed)

In this case - a 'settings' menu item appears to be missing from the sidebar for the test user.

This further indicates it's a user role/permissions item. It's unlikely that the same change would impact the site sidebar and a group page level button.

mentioned in issue gitlab-org/gitlab#396747 (closed)

added NeedsRootCause label

added NeedsService label

Hi @nmccorrison,

This issue now has the NeedsCorrectiveActions label, this label will be removed automatically when there is at least one related issue that is labeled with corrective action or ~"infradev". While it is not strictly required, having an issue related with these labels helps to ensure a similar incident doesn't happen again.

You are welcome to help improve this comment.

added NeedsCorrectiveActions label

assigned to @twk3

We are turning off admin mode in staging and rerunning the specs to see if that changes the behaviour. Meanwhile also looking for indications that this mode's enabled status recently changed or not.

Validated that specs passed with the admin_mode sign-in restriction disabled in staging, and made the same settings change in production on gitlab.com

Further investigation linked gitlab-org/gitlab!116364 (merged) and the corresponding FF being enabled with a curl API request that included admin_mode: true in the params (logs: https://log.gprd.gitlab.net/goto/dea4a4c0-da34-11ed-a017-0d32180b1390)

@dblessing also updated the description on that MR to remove the extra parameter and avoid copy/paste errors should anyone use that as a reference for re-running the command

Log for application settings in staging: https://nonprod-log.gitlab.net/goto/9e8f95e0-da37-11ed-bb50-33eb1f5eb489

Suggested corrective action regarding making changes to instance settings easier to audit.

This has also been brought up previously here: gitlab-org/gitlab#282428 (closed)

added IncidentMitigated label and removed IncidentActive label

removed blocks deployments label

@jarv Per https://log.gprd.gitlab.net/goto/24ca7a30-da33-11ed-8afc-c9851e4645c0, it appears somewhere around 13:10 UTC a request was done to update the admin_mode and openai_api_key settings for GitLab.com and staging. This enabled Admin Mode on both instances which caused QA deployment pipelines to fail and some other inconveniences on GitLab.com (Slack link internal only). The settings have been disabled on both staging and .com

Thanks @mayra-cabrera , this was done in Change issue https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8711 which looks like is the source of admin_mode. I'll followup with @mksionek to see why that was included and if we can omit it.

Thank you @jarv for tracking down that admin_mode:true was an errant copy/paste issue. I'm super glad to see it wasn't required to enable OpenAI this week.