2020-09-14 Flaky examples

This report includes several parts:

• Top 20 flaky tests sorted by occurrences per file
• Number of flaky tests per type
• Flaky tests per day for the last 20 days
• Top 20 flaky tests sorted by number of reports
• ~"failure::flaky-test" issues

Top 20 flaky tests sorted by occurrences per file

./spec/features/merge_request/user_sees_merge_widget_spec.rb: 12
./spec/features/search/user_uses_header_search_field_spec.rb: 9
./spec/support/shared_examples/features/discussion_comments_shared_example.rb: 9
./spec/support/shared_examples/features/resolving_discussions_in_issues_shared_examples.rb: 5
./ee/spec/features/analytics/cycle_analytics/cycle_analytics_spec.rb: 4
./ee/spec/features/boards/swimlanes/epics_swimlanes_filtering_spec.rb: 3
./spec/features/issues/notes_on_issues_spec.rb: 3
./spec/features/projects/compare_spec.rb: 3
./spec/models/clusters/applications/runner_spec.rb: 3
./spec/features/merge_request/user_posts_notes_spec.rb: 3
./spec/finders/labels_finder_spec.rb: 3
./ee/spec/services/ci/minutes/batch_reset_service_spec.rb: 3
./spec/support/shared_examples/models/concerns/bulk_insert_safe_shared_examples.rb: 2
./spec/features/issues/user_creates_branch_and_merge_request_spec.rb: 2
./spec/models/ci/job_artifact_spec.rb: 2
./spec/features/issues/user_comments_on_issue_spec.rb: 2
./spec/support/shared_examples/features/reportable_note_shared_examples.rb: 2
./ee/spec/features/projects/custom_projects_template_spec.rb: 2
./spec/features/broadcast_messages_spec.rb: 2
./spec/features/issuables/markdown_references/jira_spec.rb: 2

Number of flaky tests per type

features: 63
spec: 33
support: 20
models: 6
requests: 6
controllers: 5
lib: 3
finders: 3
services: 3
migrations: 2
serializers: 1
factories_spec.rb: 1

Flaky tests per day for the last 20 days

2020-09-13: 25 files
	["./spec/controllers/concerns/renders_commits_spec.rb", "./spec/features/projects/show/user_sees_git_instructions_spec.rb", "./ee/spec/services/groups/autocomplete_service_spec.rb", "./spec/controllers/projects/releases/evidences_controller_spec.rb", "./spec/features/projects/show/user_sees_collaboration_links_spec.rb", "./spec/features/security/project/public_access_spec.rb", "./spec/features/issues/move_spec.rb", "./ee/spec/features/issues/issue_sidebar_spec.rb", "./ee/spec/lib/gem_extensions/elasticsearch/model/adapter/active_record/records_spec.rb", "./ee/spec/features/markdown/metrics_spec.rb", "./spec/controllers/projects/merge_requests_controller_spec.rb", "./spec/features/merge_request/user_posts_notes_spec.rb", "./spec/support/shared_examples/features/resolving_discussions_in_issues_shared_examples.rb", "./ee/spec/features/issues/epic_in_issue_sidebar_spec.rb", "./spec/controllers/projects/pipelines_controller_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/controllers/projects/jobs_controller_spec.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./spec/features/boards/sidebar_spec.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./spec/support/shared_examples/models/concerns/bulk_insert_safe_shared_examples.rb", "./ee/spec/features/admin/geo/admin_geo_projects_spec.rb", "./spec/features/projects/files/user_creates_files_spec.rb", "./ee/spec/features/projects/custom_projects_template_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb"])
2020-09-12: 40 files
	["./spec/features/admin/admin_mode/login_spec.rb", "./ee/spec/services/groups/autocomplete_service_spec.rb", "./spec/features/security/project/internal_access_spec.rb", "./ee/spec/migrations/migrate_scim_identities_to_saml_for_new_users_spec.rb", "./ee/spec/migrations/migrate_scim_identities_to_saml_for_new_users_spec.rb", "./spec/finders/labels_finder_spec.rb", "./spec/finders/labels_finder_spec.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./ee/spec/features/search/elastic/project_search_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./ee/spec/features/search/elastic/project_search_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./spec/finders/labels_finder_spec.rb", "./spec/features/issues/note_polling_spec.rb", "./spec/features/issues/filtered_search/search_bar_spec.rb", "./spec/support/shared_examples/features/resolving_discussions_in_issues_shared_examples.rb", "./spec/features/broadcast_messages_spec.rb", "./spec/support/shared_examples/features/resolving_discussions_in_issues_shared_examples.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./ee/spec/services/dashboard/operations/list_service_spec.rb", "./spec/requests/profiles/notifications_controller_spec.rb", "./spec/features/issues/user_creates_branch_and_merge_request_spec.rb", "./spec/features/projects/compare_spec.rb", "./spec/serializers/pipeline_serializer_spec.rb", "./ee/spec/support/shared_examples/quick_actions/issuable/unassign_shared_examples.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/features/issues/notes_on_issues_spec.rb", "./spec/features/issuables/markdown_references/jira_spec.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./ee/spec/tasks/gitlab/seed/group_seed_rake_spec.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./ee/spec/features/boards/swimlanes/epics_swimlanes_filtering_spec.rb", "./spec/support/shared_examples/features/reportable_note_shared_examples.rb", "./spec/support/shared_examples/lib/gitlab/cycle_analytics/base_stage_shared_examples.rb", "./ee/spec/features/boards/swimlanes/epics_swimlanes_filtering_spec.rb", "./spec/features/issues/notes_on_issues_spec.rb", "./spec/features/triggers_spec.rb"])
2020-09-11: 22 files
	["./spec/support/shared_examples/features/resolving_discussions_in_issues_shared_examples.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/features/markdown/mermaid_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./ee/spec/features/promotion_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/services/merge_requests/create_from_issue_service_spec.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./spec/models/clusters/applications/runner_spec.rb", "./spec/requests/api/labels_spec.rb", "./spec/models/clusters/applications/runner_spec.rb", "./spec/support/shared_examples/features/reportable_note_shared_examples.rb", "./spec/features/projects/compare_spec.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./spec/features/merge_request/user_resolves_diff_notes_and_discussions_resolve_spec.rb", "./spec/support/shared_examples/models/concerns/bulk_insert_safe_shared_examples.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./ee/spec/features/boards/swimlanes/epics_swimlanes_filtering_spec.rb", "./spec/lib/gitlab/ci/build/context/global_spec.rb", "./spec/features/dashboard/datetime_on_tooltips_spec.rb", "./spec/factories_spec.rb"])
2020-09-10: 13 files
	["./ee/spec/services/ci/minutes/batch_reset_service_spec.rb", "./ee/spec/services/ci/minutes/batch_reset_service_spec.rb", "./spec/services/projects/autocomplete_service_spec.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./spec/lib/gitlab/database/rename_reserved_paths_migration/v1/rename_projects_spec.rb", "./spec/migrations/schedule_backfill_push_rules_id_in_projects_spec.rb", "./spec/requests/api/merge_requests_spec.rb", "./ee/spec/services/ci/minutes/batch_reset_service_spec.rb", "./spec/features/issues/user_creates_branch_and_merge_request_spec.rb", "./spec/features/merge_request/user_edits_merge_request_spec.rb", "./ee/spec/support/shared_examples/graphql/mutations/on_demand_scan_with_user_abilities_shared_examples.rb", "./ee/spec/features/groups/group_settings_spec.rb", "./spec/models/ci/job_artifact_spec.rb"])
2020-09-09: 13 files
	["./spec/services/projects/autocomplete_service_spec.rb", "./spec/features/dashboard/datetime_on_tooltips_spec.rb", "./spec/features/issuables/markdown_references/jira_spec.rb", "./ee/spec/features/analytics/cycle_analytics/cycle_analytics_spec.rb", "./ee/spec/features/analytics/cycle_analytics/cycle_analytics_spec.rb", "./ee/spec/features/analytics/cycle_analytics/cycle_analytics_spec.rb", "./ee/spec/features/analytics/cycle_analytics/cycle_analytics_spec.rb", "./spec/features/merge_request/user_sees_pipelines_spec.rb", "./spec/features/search/user_uses_header_search_field_spec.rb", "./spec/support/shared_examples/features/resolving_discussions_in_issues_shared_examples.rb", "./spec/features/merge_request/user_merges_when_pipeline_succeeds_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/features/projects/compare_spec.rb"])
2020-09-08: 16 files
	["./spec/features/issues/note_polling_spec.rb", "./spec/migrations/schedule_link_lfs_objects_projects_spec.rb", "./ee/spec/features/issues/form_spec.rb", "./spec/features/broadcast_messages_spec.rb", "./spec/features/issues/user_resets_their_incoming_email_token_spec.rb", "./spec/features/issues/user_comments_on_issue_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./spec/features/merge_request/user_sees_merge_widget_spec.rb", "./ee/spec/models/project_spec.rb", "./ee/spec/lib/gitlab/graphql/aggregations/vulnerabilities/lazy_user_notes_count_aggregate_spec.rb", "./spec/features/issues/notes_on_issues_spec.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./spec/features/security/project/private_access_spec.rb", "./spec/requests/api/ci/runner/jobs_request_post_spec.rb", "./spec/features/merge_request/user_posts_notes_spec.rb", "./spec/lib/gitlab/ci/build/policy/variables_spec.rb"])
2020-09-07: 17 files
	["./ee/spec/features/promotion_spec.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb", "./spec/features/merge_request/user_merges_when_pipeline_succeeds_spec.rb", "./spec/features/issues/issue_sidebar_spec.rb", "./spec/features/issues/user_edits_issue_spec.rb", "./spec/support/shared_examples/features/project_upload_files_shared_examples.rb", "./spec/features/merge_request/user_posts_notes_spec.rb", "./spec/requests/api/jobs_spec.rb", "./ee/spec/features/projects/custom_projects_template_spec.rb", "./spec/features/dashboard/projects_spec.rb", "./spec/features/issues/user_comments_on_issue_spec.rb", "./spec/models/ci/job_artifact_spec.rb", "./spec/requests/api/groups_spec.rb", "./spec/models/clusters/applications/runner_spec.rb", "./spec/models/deployment_metrics_spec.rb", "./spec/support/shared_examples/features/discussion_comments_shared_example.rb"])

Top 20 flaky tests sorted by number of reports

./spec/controllers/concerns/renders_commits_spec.rb: 2006
./spec/features/projects/show/user_sees_git_instructions_spec.rb: 1569
./spec/controllers/projects/releases/evidences_controller_spec.rb: 482
./spec/features/projects/show/user_sees_collaboration_links_spec.rb: 480
./spec/features/security/project/public_access_spec.rb: 280
./spec/features/security/project/internal_access_spec.rb: 181
./ee/spec/services/groups/autocomplete_service_spec.rb: 176
./spec/controllers/projects/merge_requests_controller_spec.rb: 130
./spec/features/issues/note_polling_spec.rb: 127
./spec/features/admin/admin_mode/login_spec.rb: 115
./ee/spec/features/issues/issue_sidebar_spec.rb: 104
./ee/spec/features/markdown/metrics_spec.rb: 73
./ee/spec/services/ci/minutes/batch_reset_service_spec.rb: 66
./spec/services/projects/autocomplete_service_spec.rb: 64
./ee/spec/lib/gem_extensions/elasticsearch/model/adapter/active_record/records_spec.rb: 61
./spec/features/merge_request/user_sees_merge_widget_spec.rb: 56
./spec/controllers/projects/jobs_controller_spec.rb: 50
./ee/spec/migrations/migrate_scim_identities_to_saml_for_new_users_spec.rb: 46
./spec/features/issues/move_spec.rb: 43
./spec/support/shared_examples/features/discussion_comments_shared_example.rb: 39

failureflaky-test issues

• gitlab-org/gitlab#247120 (closed) Broken master: ./ee/spec/models/concerns/epic_tree_sorting_spec.rb:22 # EpicTreeSorting#move_after moves an epic_issue ~"Category:Epics", devopsplan, epic tree, failureflaky-test, ~"feature", ~"feature::maintenance", sectiondev, test
• gitlab-org/gitlab#246794 (closed) Flaky spec in spec/lib/gitlab/search_results_spec.rb | Gitlab::SearchResults as a user with access #objects uses page and per_page to paginate results Category:Global Search, ~"devops::enablement", failureflaky-test, groupglobal search, ~"section::enablement", test
• gitlab-org/gitlab#246765 (closed) Long Jest Spec at spec/frontend/ci_variable_list/ci_variable_list/ajax_variable_list_spec.js | AjaxFormVariableList › updateRowsWithPersistedVariables › updates new variable row with persisted ID ~"ci::variables", devopsverify, failureflaky-test, frontend, ~"group::continuous integration", sectionops, test
• gitlab-org/gitlab#240932 (closed) Flaky test in spec/features/boards/sidebar_spec.rb:158 ~"Category:Issue Boards", Engineering Productivity, devopsplan, failureflaky-test, groupproject management, sectiondev, severity3, test
• gitlab-org/gitlab#234081 Flaky test spec/features/markdown/mermaid_spec.rb:47 Engineering Productivity, Quality, devopsplan, failureflaky-test, groupproject management, markdown, sectiondev, severity2, test, workflowready for development
• gitlab-org/gitlab#233464 (closed) Flaky spec in spec/frontend/monitoring/components/charts/stacked_column_spec.js | Stacked column chart component › legend › when inline legend layout prop is set › should render an inline legend layout Engineering Productivity, Quality, devopsmonitor, failureflaky-test, frontend, ~"group::apm", sectionops, severity3, test
• gitlab-org/gitlab#233376 (closed) Flaky spec: spec/features/issues/user_uses_quick_actions_spec.rb devopsplan, failureflaky-test, groupproject management, priority3, sectiondev, severity3, test
• gitlab-org/gitlab#233112 (closed) Transient failure in spec/features/issues/gfm_autocomplete_spec.rb | GFM autocomplete when tribute_autocomplete feature flag is on when other notes are destroyed keeps autocomplete key listeners backend, devopsplan, failureflaky-test, groupproject management, markdown, sectiondev, severity3, test
• gitlab-org/gitlab#232594 (closed) Flaky: issue move to another project service desk issue moved to a project with service desk disabled shows an alert after being moved devopscreate, failureflaky-test, groupsource code, issues, priority2, sectiondev, severity2
• gitlab-org/gitlab#231431 (closed) Flaky test spec/features/issues/gfm_autocomplete_spec.rb:652 Quality, devopsplan, failureflaky-test, groupproject management, sectiondev, severity2
• gitlab-org/gitlab#228683 (closed) spec/frontend/environments/environments_app_spec.js slow Category:Release Orchestration, ~"devops::release", environments, failureflaky-test, frontend, ~"group::progressive delivery", priority2, sectionops, severity2
• gitlab-org/gitlab#227845 (closed) Flaky test: /ee/spec/controllers/ee/projects/jobs_controller_spec.rb:65 ~"Accepting merge requests", Category:Continuous Integration, Quality, ~"ci::processing", devopsverify, failureflaky-test, found:in MR, ~"group::continuous integration", sectionops, severity2, test
• gitlab-org/gitlab#225466 (closed) Flaky test spec/requests/api/files_spec.rb:834 # API::Files POST /projects/:id/repository/files with binary file remains unchanged Engineering Productivity, api, devopscreate, failureflaky-test, groupsource code, priority3, repository, sectiondev
• gitlab-org/gitlab#220894 (closed) Job Failed spec/controllers/projects/blob_controller_spec.rb:376 Quality, devopscreate, failureflaky-test, found:master, groupsource code, priority3, sectiondev, severity3
• gitlab-org/gitlab#218642 (closed) Transient Failure in spec/features/issues/note_polling_spec.rb | Issue notes polling updates when from own user when editing but you changed some things, an update comes in, and you press cancel, show the updated content Engineering Productivity, backstage [DEPRECATED], devopsmanage, failureflaky-test, groupproject management, issues, priority2, sectiondev, severity3, test
• gitlab-org/gitlab#218534 (closed) Transient failures in spec/features/discussion_comments/snippets_spec.rb | Thread Comments Snippet behaves like thread comments when the toggle is clicked when selecting "Start thread" when opening the menu has "Start thread" selected ~"Category:Snippets", Engineering Productivity, backstage [DEPRECATED], devopscreate, failureflaky-test, ~"group::editor", sectiondev, test
• gitlab-org/gitlab#217551 (closed) Transient failures in spec/features/projects/compare_spec.rb | Compare tags compares tags devopscreate, failureflaky-test, groupsource code, priority3, repository, sectiondev, severity3
• gitlab-org/gitlab#217550 (closed) Transient failures in spec/features/issues/user_sorts_issue_comments_spec.rb | Comment sort direction on issue page saves sort order devopsplan, failureflaky-test, groupproject management, issues, priority3, sectiondev, severity3
• gitlab-org/gitlab#215611 (closed) Transient failure in spec/features/projects/compare_spec.rb:80 # Compare branches when project have an open merge request compares branches ~"Accepting merge requests", Engineering Productivity, ~"bug", ci-build, devopscreate, failureflaky-test, groupsource code, ~"missed-SLO", priority2, sectiondev, severity2, test
• gitlab-org/gitlab#213205 (closed) Flaky test './spec/features/projects/files/user_uploads_files_spec.rb[1:1:2]' # Projects > Files > User uploads files when a user has write access uploads and commit a new image file Engineering Productivity, devopsplan, failureflaky-test, groupproject management, priority2, sectiondev, severity2
• gitlab-org/gitlab#208266 (closed) Flaky test in spec/controllers/projects/pipelines_controller_spec.rb:91 # Projects::PipelinesController GET index.json when using legacy stages does not execute N+1 queries ~"Accepting merge requests", Category:Continuous Integration, Engineering Productivity, backend, ~"ci::processing", devopsverify, failureflaky-test, ~"group::continuous integration", priority2, sectionops, severity2, test
• gitlab-org/gitlab#207794 (closed) Transient job failure with could not lock config file /builds/gitlab-org/gitlab.tmp/git-template/config: File exists Engineering Productivity, eppipeline, failureflaky-test, priority2, severity2
• gitlab-org/gitlab#199070 (closed) Transient failure in spec/features/projects/compare_spec.rb / Selenium::WebDriver::Error::StaleElementReferenceError ~"Accepting merge requests", Engineering Productivity, Quality, backend, ~"bug", devopscreate, failureflaky-test, groupsource code, ~"missed-SLO", missed:12.10, missed:13.0, priority2, repository, sectiondev, severity2, test, ~""
• gitlab-org/gitlab#198691 (closed) Flaky Spec: User rebases a merge request when merge is regular behaves like rebases rebases Engineering Productivity, Quality, devopscreate, failureflaky-test, groupsource code, priority3, sectiondev, severity3, test
• gitlab-org/gitlab#197876 (closed) CI Job Failed - no space left on device (executor_docker.go:745:0s) ~"Category:Runner", devopsverify, failureflaky-test, gitlab.com, grouprunner, infradev, missed:12.10, missed:12.8, missed:12.9, missed:13.0, missed:13.3, pipeline, priority2, sectionops, severity2, ~"workflow::In dev"
• gitlab-org/gitlab#196516 (closed) Transient failure in spec/features/issues/form_spec.rb devopsplan, failureflaky-test, groupproject management, issues, priority3, sectiondev, severity3
• gitlab-org/gitlab#196164 (closed) Master failure spec/features/discussion_comments/merge_request_spec.rb:17 devopscreate, failureflaky-test, groupsource code, merge requests, priority3, sectiondev
• gitlab-org/gitlab#32713 (closed) Flaky test in master: /spec/features/discussion_comments/issue_spec.rb Quality, devopscreate, failureflaky-test, groupsource code, issues, sectiondev
• gitlab-org/gitlab#30682 (closed) Flaky spec spec/features/projects/commit/user_reverts_commit_spec.rb:35 Create [DEPRECATED], devopscreate, failureflaky-test, groupsource code, merge requests, priority2, sectiondev, severity2
• gitlab-org/gitlab#30160 (closed) Rspec spawns many gitaly processes in a loop Category:GDK, Quality, failureflaky-test, ~"group::ecosystem", sectiondev
• gitlab-org/gitlab#30150 (closed) Flaky spec: Too many SQL queries were executed in Projects::MergeRequestsController#show Quality, devopscreate, failureflaky-test, groupsource code, sectiondev, severity3
• gitlab-org/gitlab#29489 (closed) Flaky spec in spec/features/issues/notes_on_issues_spec.rb - ERROR: Job failed: execution took longer than 1h40m0s seconds Quality, devopscreate, failureflaky-test, groupsource code, priority3, sectiondev, severity3, test
• gitlab-org/gitlab#29429 (closed) Flaky spec - spec/features/merge_request/user_resolves_diff_notes_and_discussions_resolve_spec.rb:180 Quality, failureflaky-test, groupsource code, sectiondev, severity3, test
• gitlab-org/gitlab#11951 (closed) [BUG] Segmentation fault at 0x0000000000000000 Engineering Productivity, Enterprise Edition, Quality, failureflaky-test, test
• gitlab-org/gitlab#28166 (closed) Look into slow Jest tests Quality, failureflaky-test, frontend, test
• gitlab-org/gitlab#27965 (closed) Flaky spec: "list of badges user changes current ref of build status badge " / spec/features/projects/badges/list_spec.rb:43 ~"Accepting merge requests", Category:Continuous Integration, Quality, ~"bug", ci-build, ~"ci::processing", devopsverify, failureflaky-test, frontend, ~"group::continuous integration", priority3, sectionops, severity3, test

added Engineering Productivity triage report labels

mentioned in issue #382 (closed)

mentioned in issue #372 (closed)

I added test stage/group labels to the issues and cleaned up a few older issues.

unassigned @kwiebers

mentioned in issue #393 (closed)

Closing in favor of #392 (closed).

closed