Spinach builds failing but marked as successful
NOTE: This is an issue with Spinach, not CI, as far as I can tell.
While working on the Rails 5 MR (!5555 (closed)) I've noticed that the Spinach builds will sometimes error out but then return an exit value that makes CI think they've passed. I don't know if we can fix this on our end, or if this is something that'd need to be fixed with a patch to the gem itself (good luck with that). If this is something we can fix easily, we should. Otherwise it's probably not enough of a problem to be concerning (Rspec usually has tests that would also throw errors if Spinach were, and they'd fail properly).
I haven't seen it effect any other MRs, but I figured I'd report it before it has the chance to bite us later on.
Examples:
- Many of the passing Spinach builds for this pipeline: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/4330647
- https://gitlab.com/gitlab-org/gitlab-ce/builds/4304365
- https://gitlab.com/gitlab-org/gitlab-ce/builds/4304372