Above all else, please remember that the aim of pipeline triage is to identify problems and try to get them resolved (ideally) before they impact users.
Failure in browser_ui/9_data_stores/group/transfer_project_spec.rb | Data Stores Project transfer between groups user transfers a project between groups
Failure in ee/browser_ui/10_govern/create_merge_request_with_secure_spec.rb | Govern Security Reports in a Merge Request Widget displays vulnerabilities in merge request widget
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an auto-remediation MR from mr security widget
Failure in ee/browser_ui/4_verify/pipelines_for_merged_results_and_merge_trains_spec.rb | Verify Pipelines for merged results and merge trains merges via a merge train
I closed many issues relating to failures in main and staging from gitlab-com/gl-infra/production#14259 (closed) but there may be some still open because the issue affected pre and prod also.
Failure in browser_ui/4_verify/ci_variable/pipeline_with_protected_variable_spec.rb | Verify Pipeline with protected variable does not expose variable on unprotected branch
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an issue from a vulnerability from mr security widget
Failure in ee/browser_ui/11_fulfillment/purchase/purchase_storage_spec.rb | Fulfillment without active subscription behaves like Purchase storage adds additional storage to group namespace
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin sets user cap limit for group shows members over limit as pending for approvals
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for group invite when inviting a group with developer role which increases seats owed behaves like overage for group invite shows...
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin sets user cap limit for group shows members over limit as pending for approvals
You may occasionally notice errors like the following in Staging:
An unexpected error occurred while reporting test results in issues.The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/10051106Gitlab::Error::Forbidden Server responded with code 403, message: 403 Forbidden. Request URI: https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/issues/403189/notes/1334183447
FYI the change to Postgres version 13.9 breaks tests on GDK until the image is updated:
ERROR: PostgreSQL data directory is version 12 and must be upgraded to version 13.9 before GDK can be updated.Run 'support/upgrade-postgresql' to back up and upgrade the PostgreSQL data directory.
Failure in ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted protected branch push and merge when only one user is allowed to merge and push to a protected branch behaves like selected developer user pushes an...
Failure in browser_ui/5_package/container_registry/container_registry_omnibus_spec.rb | Package Self-managed Container Registry when tls is disabled using docker:20.10 and a personal access token pushes image and deletes tag
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for group invite does not show overage modal when inviting a group which does not increase seats owed
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for group invite when inviting a group with developer role which increases seats owed behaves like overage for group invite shows...
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for member invite when access level developer or above is added behaves like overage for member invite shows the modal
Failure in browser_ui/9_data_stores/group/transfer_project_spec.rb | Data Stores Project transfer between groups user transfers a project between groups
Failure in browser_ui/5_package/container_registry/container_registry_omnibus_spec.rb | Package Self-managed Container Registry when tls is disabled using docker:20.10 and a personal access token pushes image and deletes tag
Failure in ee/browser_ui/4_verify/pipelines_for_merged_results_and_merge_trains_spec.rb | Verify Pipelines for merged results and merge trains merges via a merge train
Failure in ee/browser_ui/10_govern/create_merge_request_with_secure_spec.rb | Govern Security Reports in a Merge Request Widget displays vulnerabilities in merge request widget
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an auto-remediation MR from mr security widget
Failure in ee/browser_ui/11_fulfillment/utilization/usage_quotas_seats_spec.rb | Fulfillment Utilization in usage quotas user seat data is displayed correctly
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for member invite when access level developer or above is added behaves like overage for member invite shows the modal
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for group invite does not show overage modal when inviting a group which does not increase seats owed
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for member invite does not show overage modal when inviting a member as a guest
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin removes user cap limit for group does not automatically approve pending members
Failure in browser_ui/3_create/source_editor/source_editor_toolbar_spec.rb | Create Source editor toolbar preview can preview markdown side-by-side while editing
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin sets user cap limit for group shows members over limit as pending for approvals
Failure in browser_ui/5_package/container_registry/container_registry_omnibus_spec.rb | Package Self-managed Container Registry when tls is disabled using docker:20.10 and a personal access token pushes image and deletes tag
The PostgreSQL upgrade for Staging needed to be rolled back today since several reliable and smoke E2E tests had failed with various errors such as 404s, 403s, 500s, etc., revealing performance issues.
Due to being on the PG upgrade call most of my day, I did not get a chance to look into gitlab-org/gitlab#411583 (closed) or the Staging Ref qa-reliable failures mentioned above. @jay_mccure@nwestbury if you happen to have some time during your shift and would be able to look into those, that would be awesome
Failure in browser_ui/3_create/source_editor/source_editor_toolbar_spec.rb | Create Source editor toolbar preview can preview markdown side-by-side while editing
Failure in browser_ui/8_monitor/alert_management/alert_settings_create_new_alerts_spec.rb | Monitor Alert settings when using Prometheus integration behaves like sends test alert creates new alert
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an auto-remediation MR from mr security widget
Failure in api/12_systems/gitaly/changing_repository_storage_spec.rb | Systems Changing Gitaly repository storage when moving from Gitaly to Gitaly Cluster behaves like repository storage move confirms a finished status after moving project repositor...
Failure in ee/browser_ui/3_create/repository/code_owners_with_protected_branch_and_squashed_commits_spec.rb | Create Setup an MR with codeowners file creates a merge request with codeowners file and squashing commits enabled
Failure in ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted protected branch push and merge when only one group is allowed to merge and push to a protected branch behaves like selected developer user pushes a...
Failure in browser_ui/3_create/merge_request/suggestions/custom_commit_suggestion_spec.rb | Create with merge request suggestions applies a single suggestion with a custom message
Failure in api/4_verify/api_variable_inheritance_with_forward_pipeline_variables_spec.rb | Verify Pipeline API defined variable inheritance is determined based on forward:pipeline_variables condition
Failure in ee/browser_ui/4_verify/pipeline_subscription_with_group_owned_project_spec.rb | Verify Pipeline subscription with a group owned project when upstream project new tag pipeline finishes triggers pipeline in downstream project
Failure in browser_ui/3_create/source_editor/source_editor_toolbar_spec.rb | Create Source editor toolbar preview can preview markdown side-by-side while editing
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for group invite does not show overage modal when inviting a group which does not increase seats owed
Failure in ee/browser_ui/11_fulfillment/utilization/usage_quotas_seats_spec.rb | Fulfillment Utilization in usage quotas user seat data is displayed correctly
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for member invite when access level developer or above is added behaves like overage for member invite shows the modal
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with ultimate plan for member invite does not show overage modal when inviting a member as a guest
Failure in browser_ui/5_package/container_registry/container_registry_omnibus_spec.rb | Package Self-managed Container Registry when tls is disabled using docker:20.10 and a personal access token pushes image and deletes tag
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an issue from a security finding in pipeline security tab
Failure in ee/browser_ui/10_govern/change_vulnerability_status_spec.rb | Govern project vulnerability report can successfully bulk change status in vulnerability report
Failure in api/12_systems/gitaly/changing_repository_storage_spec.rb | Systems Changing Gitaly repository storage when moving from Gitaly to Gitaly Cluster behaves like repository storage move confirms a finished status after moving project repositor...
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an issue from a vulnerability from mr security widget
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an auto-remediation MR from mr security widget
Failure in browser_ui/3_create/source_editor/source_editor_toolbar_spec.rb | Create Source editor toolbar preview can preview markdown side-by-side while editing
Almost all reliable and smoke tests in Production were failing for about an hour or so with GitLab: Internal API unreachable. errors. This was caused by gitlab-com/gl-infra/production#14422 (closed), which is now resolved. Hopefully I was able to bulk close all the related issues to this
Several failures are awaiting on various fixes or quarantines:
Failure in browser_ui/3_create/source_editor/source_editor_toolbar_spec.rb | Create Source editor toolbar preview can preview markdown side-by-side while editing
Failure in browser_ui/3_create/source_editor/source_editor_toolbar_spec.rb | Create Source editor toolbar preview can preview markdown side-by-side while editing
Failure in browser_ui/5_package/container_registry/container_registry_omnibus_spec.rb | Package Self-managed Container Registry when tls is disabled using docker:18.09.9 and a project deploy token pushes image and deletes tag
Failure in browser_ui/5_package/container_registry/container_registry_omnibus_spec.rb | Package Self-managed Container Registry when tls is disabled using docker:20.10 and a personal access token pushes image and deletes tag
Failure in ee/browser_ui/5_package/dependency_proxy_sso_spec.rb | Package Dependency Proxy Group SSO pulls an image using the dependency proxy on a group enforced SSO
Failure in browser_ui/3_create/source_editor/source_editor_toolbar_spec.rb | Create Source editor toolbar preview can preview markdown side-by-side while editing
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin removes user cap limit for group does not automatically approve pending members
Failure in ee/browser_ui/2_plan/epic/epics_management_spec.rb | Plan Epics Management Resources created via API Visit epic first closes and reopens an epic
Failure in ee/browser_ui/15_growth/free_trial_spec.rb | Growth SaaS trials starts a free trial when on about page with multiple eligible namespaces registers for a new trial
Failure in browser_ui/3_create/snippet/add_comment_to_snippet_spec.rb | Create Adding comments on snippets behaves like comments on snippets adds, edits, and deletes a comment on a project_snippet
Failure in ee/browser_ui/5_package/dependency_proxy_sso_spec.rb | Package Dependency Proxy Group SSO pulls an image using the dependency proxy on a group enforced SSO
Failure in ee/browser_ui/13_secure/enable_scanning_from_configuration_spec.rb | Secure Enable Scanning from UI enable sast from configuration runs sast job when enabled from configuration
Failure in ee/browser_ui/13_secure/enable_scanning_from_configuration_spec.rb | Secure Enable Scanning from UI enable dependency scanning from configuration runs dependency scanning job when enabled from configuration
The PostgreSQL upgrade team is planning to do a test on Staging sometime today, and may require assistance from the DRI who is online to run the smoke tests and look into any failures. They will let us know when they plan to do that: gitlab-com/gl-infra/production#9630 (comment 1396499362)
Multiple failures due to some mismatched state on environments when the super sidebar was enabled for all tests today: gitlab-org/gitlab#412139 (closed). This should now be resolved.
Failure in ee/browser_ui/5_package/dependency_proxy_sso_spec.rb | Package Dependency Proxy Group SSO pulls an image using the dependency proxy on a group enforced SSO
Failure in browser_ui/3_create/repository/push_mirroring_lfs_over_http_spec.rb | Create Push mirror a repository over HTTP configures and syncs LFS objects for a (push) mirrored repository
Failure in browser_ui/3_create/repository/push_mirroring_over_http_spec.rb | Create Push mirror a repository over HTTP configures and syncs a (push) mirrored repository
Failure in browser_ui/3_create/repository/push_mirroring_lfs_over_http_spec.rb | Create Push mirror a repository over HTTP configures and syncs LFS objects for a (push) mirrored repository
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an auto-remediation MR from mr security widget
Failure in ee/browser_ui/13_secure/enable_scanning_from_configuration_spec.rb | Secure Enable Scanning from UI enable dependency scanning from configuration runs dependency scanning job when enabled from configuration
Failure in ee/browser_ui/13_secure/enable_scanning_from_configuration_spec.rb | Secure Enable Scanning from UI enable sast from configuration runs sast job when enabled from configuration
Failure in browser_ui/5_package/container_registry/container_registry_omnibus_spec.rb | Package Self-managed Container Registry when tls is disabled using docker:18.09.9 and a project deploy token pushes image and deletes tag
Failure in browser_ui/3_create/snippet/add_comment_to_snippet_spec.rb | Create Adding comments on snippets behaves like comments on snippets adds, edits, and deletes a comment on a project_snippet
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with premium plan for member invite when guest role is added behaves like overage for member invite shows the modal
Failure in ee/browser_ui/3_create/repository/project_templates_spec.rb | Create Project templates when instance level successfully imports the project using template
Failure in ee/browser_ui/10_govern/group/group_compliance_dashboard_spec.rb | Govern compliance dashboard with separation of duties in an MR when there is only one approval from a user other than the author shows only less than two approvers violation
Failure in ee/browser_ui/10_govern/scan_result_policy_vulnerabilities_spec.rb | Govern Scan result policy requires approval when a pipeline report has findings matching the scan result policy
Failure in ee/browser_ui/10_govern/group/group_audit_logs_1_spec.rb | Govern Group for add and remove project access behaves like audit event logs audit events for UI operations
Failure in ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb | Fulfillment Seat overage modal with premium plan for member invite when guest role is added behaves like overage for member invite shows the modal
Failure in browser_ui/3_create/snippet/add_comment_to_snippet_spec.rb | Create Adding comments on snippets behaves like comments on snippets adds, edits, and deletes a comment on a project_snippet
There was also mention of a test they wanted to perform in Staging yesterday, but it doesn't look like this happened yet, and they may need Quality to kick off smoke tests / look over any QA failures. I've asked them to notify the #quality channel as well, when that does happen: gitlab-com/gl-infra/production#9630 (comment 1396499362)
Please also see the handbook for what is expected from QA DRIs during this upgrade
Just wanted to let you all know so no one is caught off guard like I was