Skip to content

Add flaky report by example_id

Albert Salim requested to merge add-example-id-to-flaky-report into master

Add example id to flaky report. Some test files contain a very large number of tests. Having the example id would help to pinpoint the flaky test among many other tests in the file.


### Top 20 flaky test examples sorted by occurrences per example

./spec/models/ci/job_artifact_spec.rb[1:10:2:1]: 2
./spec/features/projects/pipelines/pipelines_spec.rb[1:1:1:10:2]: 1
./spec/features/search/user_uses_header_search_field_spec.rb[1:1:5]: 1
./spec/features/merge_request/user_posts_notes_spec.rb[1:2:2:1]: 1
./ee/spec/lib/ee/gitlab/background_migration/update_location_fingerprint_for_container_scanning_findings_spec.rb[1:2]: 1
./spec/features/merge_request/user_sees_merge_widget_spec.rb[1:20:3:6:2:1]: 1
./ee/spec/features/projects/custom_projects_template_spec.rb[1:1:1:7]: 1
./spec/features/projects/files/user_creates_files_spec.rb[1:2:1:7]: 1
./ee/spec/features/admin/geo/admin_geo_projects_spec.rb[1:1:1]: 1
./spec/features/triggers_spec.rb[1:3:2]: 1
./spec/features/issues/notes_on_issues_spec.rb[1:6:1:1]: 1
./ee/spec/features/boards/swimlanes/epics_swimlanes_filtering_spec.rb[1:1:4]: 1
./spec/lib/gitlab/cycle_analytics/code_stage_spec.rb[1:2:1:1]: 1
./spec/features/reportable_note/commit_spec.rb[1:1:1:4]: 1
./ee/spec/features/boards/swimlanes/epics_swimlanes_filtering_spec.rb[1:1:3]: 1
./spec/features/search/user_uses_header_search_field_spec.rb[1:2:1:3:2:1]: 1
./ee/spec/tasks/gitlab/seed/group_seed_rake_spec.rb[1:1]: 1
./spec/features/search/user_uses_header_search_field_spec.rb[1:1:3:1]: 1
./spec/features/issuables/markdown_references/jira_spec.rb[1:1:3:1]: 1
./spec/features/issues/notes_on_issues_spec.rb[1:3:1:1]: 1

### Top 20 flaky test examples sorted by number of reports

./spec/controllers/concerns/renders_commits_spec.rb[1:2:1]: 2008
./spec/features/projects/show/user_sees_git_instructions_spec.rb[1:1:3:2:1:1]: 1571
./spec/controllers/projects/releases/evidences_controller_spec.rb[1:1:3:2:5:1:1]: 484
./spec/features/projects/show/user_sees_collaboration_links_spec.rb[1:1:1]: 482
./spec/features/security/project/public_access_spec.rb[1:2:1]: 281
./spec/features/security/project/internal_access_spec.rb[1:2:1]: 181
./spec/controllers/projects/merge_requests_controller_spec.rb[1:8:2:3:1:1:1]: 132
./spec/features/issues/note_polling_spec.rb[1:2:1:3]: 124
./spec/features/admin/admin_mode/login_spec.rb[1:1:1:1:1]: 115
./ee/spec/services/groups/autocomplete_service_spec.rb[1:1:1]: 113
./ee/spec/features/issues/issue_sidebar_spec.rb[1:1:2]: 104
./ee/spec/features/markdown/metrics_spec.rb[1:1:2:1]: 75
./ee/spec/services/groups/autocomplete_service_spec.rb[1:1:2:1]: 64
./ee/spec/lib/gem_extensions/elasticsearch/model/adapter/active_record/records_spec.rb[1:1:1]: 61
./spec/services/projects/autocomplete_service_spec.rb[1:3:2:1]: 56
./spec/controllers/projects/jobs_controller_spec.rb[1:2:2:4:1:1]: 52
./ee/spec/services/ci/minutes/batch_reset_service_spec.rb[1:1:1:2:1]: 43
./spec/features/issues/move_spec.rb[1:3:2]: 43
./ee/spec/migrations/migrate_scim_identities_to_saml_for_new_users_spec.rb[1:1:1:2]: 33
./spec/features/issues/create_issue_for_single_discussion_in_merge_request_spec.rb[1:1:4:2:5]: 23

Action items

  • (If applicable) Add documentation to the handbook pages for Triage Operations =>
  • (If applicable) Identify the affected groups and how to communicate to them:
    • /cc @person_or_group =>
    • Relevant Slack channels =>
    • Week-in-review

Merge request reports