E2E Test Reporting Broken Due to 404 File Not Found Errors
Summary
E2E test reporting is currently broken in live environment pipelines (ex: Staging) due to 404 File Not Found
errors, which is preventing failure issues from being automatically generated.
Example job: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/14426268#L518
Example error:
/builds/gitlab-org/quality/staging/.gems/gems/gitlab-4.19.0/lib/gitlab/request.rb:71:in `validate': Server responded with code 404, message: {"message"=>"404 File Not Found"}. Request URI: https://gitlab.com/api/v4/projects/gitlab-org%2Fquality%2Ftestcases/repository/files/qa%2Fspecs%2Ffeatures%2Fee%2Fbrowser_ui%2F10_govern%2Fscan_execution_policy_vulnerabilities_spec.rb/raw (Gitlab::Error::NotFound)
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab-4.19.0/lib/gitlab/request.rb:55:in `block (2 levels) in <class:Request>'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab-4.19.0/lib/gitlab/client/repository_files.rb:20:in `file_contents'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/gitlab_client/repository_files_client.rb:17:in `block in file_contents'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/gitlab_client/gitlab_client.rb:19:in `handle_gitlab_client_exceptions'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/gitlab_client/repository_files_client.rb:16:in `file_contents'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/gitlab_client/repository_files_client.rb:22:in `file_contents_at_line'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/test_result/base_test_result.rb:71:in `calls_shared_examples?'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/report_as_issue.rb:159:in `up_to_date_labels'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/concerns/results_reporter.rb:43:in `up_to_date_labels'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/report_as_issue.rb:140:in `update_labels'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/results_in_test_cases.rb:33:in `update_testcase'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/report_results.rb:65:in `report_test'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/report_results.rb:48:in `block (2 levels) in run!'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/test_results/base_test_results.rb:21:in `each'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/test_results/base_test_results.rb:21:in `each'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/report_results.rb:43:in `block in run!'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/test_results/builder.rb:28:in `block in test_results_per_file'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/test_results/builder.rb:15:in `each'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/test_results/builder.rb:15:in `test_results_per_file'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/report_results.rb:40:in `run!'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/lib/gitlab_quality/test_tooling/report/report_as_issue.rb:23:in `invoke!'
from /builds/gitlab-org/quality/staging/.gems/gems/gitlab_quality-test_tooling-1.29.1/exe/report-results:58:in `<top (required)>'
from /builds/gitlab-org/quality/staging/.gems/bin/report-results:25:in `load'
from /builds/gitlab-org/quality/staging/.gems/bin/report-results:25:in `<main>'
Root Cause
This appears to be caused by the method calls_shared_examples?
introduced in gitlab-org/ruby/gems/gitlab_quality-test_tooling!213 (merged) pointing to the wrong project when it calls on files_client.file_contents_at_line(line_number)
For example, in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/14426268#L518, it is trying to find https://gitlab.com/api/v4/projects/gitlab-org%2Fquality%2Ftestcases/repository/files/qa%2Fspecs%2Ffeatures%2Fee%2Fbrowser_ui%2F10_govern%2Fscan_execution_policy_vulnerabilities_spec.rb/raw
It is using gitlab-org/quality/testcases
as the project to try and find the test qa/specs/features/ee/browser_ui/10_govern/scan_execution_policy_vulnerabilities_spec.rb
, but that exists in gitlab-org/gitlab