Error happening outside of RSpec examples but no error messages in job log
This was discovered from Slack (internal) and here are 2 examples of such jobs:
- https://gitlab.com/gitlab-org/gitlab/-/jobs/4410617301
- https://gitlab.com/gitlab-org/gitlab/-/jobs/4411268757
The job output ending like this:
Finished in 32 minutes 2 seconds (files took 43.17 seconds to load)
3678 examples, 0 failures, 22 pending, 1 error occurred outside of examples
Randomized with seed 12320
[TEST PROF INFO] Time spent in factories: 11:41.141 (36.37% of total time)
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
RSpec exited with 2.
Not retrying failing examples since there were errors happening outside of the RSpec examples!
Looking above there are no relevant errors. However, if we look at the artifacts RSpec produced, we can see that. For example:
It's not very readable but we can see things like:
"\n1st Try error in ./spec/models/ci/runner_spec.rb:499:\n\nexpected:
And also follows:
"\nRSpec::Retry: 2nd try ./spec/models/ci/runner_spec.rb:499","\nAn error occurred in an `after(:context)` hook.\nFailure/Error: connection.public_send(...)\n\nActiveRecord::StatementInvalid:\n PG::SREModifyingSqlDataNotPermitted: ERROR: Table: \"p_ci_runner_machine_builds\" is write protected within this Gitlab database.\n HINT: Make sure you are using the right database connection\n CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE\n# ./lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'\n# ./lib/gitlab/database/load_balancing/connection_proxy.rb:127
The second error looks like it's caused by the first error, meaning that rspec-retry didn't work well in this case, and the first error should be what we're looking for, which seems to be happening in the RSpec examples. It's the second retry causing it to have an error outside of the examples.
The first error looks like:
expected: [#Ci::Runner id: 32, ...]\n got: #ActiveRecord::Relation [Ci::Runner id: 34, 32, 35]
After removing noises, it looks like the runner id doesn't match.
Edited by Lin Jen-Shin