`rspec-retry` is biting us when some API specs fail
We've been bitten in the past with a tricky bug that would happen only on CI:
Imagine we have the following spec:
get api("/projects/#{project.id}", admin)
expect(json_response['id']).to eq(project.id)
expect(json_response['builds_enabled']).to be_present
Let's imagine json_response['id'] == 1 here.
- The API spec fails for the first time, for instance on
expect(json_response['builds_enabled']).to be_presentbecause the field is nowjobs_response. This is a legit failure, and that's the one we should see at the end of the CI run. -
rspec-retrykicks-in (because we setRSPEC_RETRY_RETRY_COUNT: "3"in our.gitlab-ci.yml) - Unfortunately,
rspec-retrydoesn't rerun the example with a clean slate (it only clearletvariables), meaning that ourjson_responsemethod will still return the memoized response from the first run, including the ID of the resource from the first run (i.e.json_response['id'] == 1)... - Thus, on the second run of the spec,
expect(json_response['id']).to eq(project.id)will fail becauseprojectwhich wasletwas cleared, thus we have a new record, with an ID of2so our expectation is nowexpect(1).to eq(2)=> FAIL -
rspec-retryretries two more times, before failing with the following failure:
Failure/Error: expect(json_response['id']).to eq(project.id)
expected: 2
got: 1
(compared using ==)
At this point you're trying to understand what's going on.
Anyways, the easiest solution to this weird bug is to implement json_response as a let variable so that it's cleared on each retries.
Edited by Brett Walker