GitLab Next

We've been bitten in the past with a tricky bug that would happen only on CI:

Imagine we have the following spec:

get api("/projects/#{project.id}", admin)

expect(json_response['id']).to eq(project.id)
expect(json_response['builds_enabled']).to be_present

Let's imagine json_response['id'] == 1 here.

1. The API spec fails for the first time, for instance on expect(json_response['builds_enabled']).to be_present because the field is now jobs_response. This is a legit failure, and that's the one we should see at the end of the CI run.
2. rspec-retry kicks-in (because we set RSPEC_RETRY_RETRY_COUNT: "3" in our .gitlab-ci.yml)
3. Unfortunately, rspec-retry doesn't rerun the example with a clean slate (it only clear let variables), meaning that our json_response method will still return the memoized response from the first run, including the ID of the resource from the first run (i.e. json_response['id'] == 1)...
4. Thus, on the second run of the spec, expect(json_response['id']).to eq(project.id) will fail because project which was let was cleared, thus we have a new record, with an ID of 2 so our expectation is now expect(1).to eq(2) => FAIL
5. rspec-retry retries two more times, before failing with the following failure:

Failure/Error: expect(json_response['id']).to eq(project.id)

       expected: 2
            got: 1

       (compared using ==)

At this point you're trying to understand what's going on.

Anyways, the easiest solution to this weird bug is to implement json_response as a let variable so that it's cleared on each retries.