e2e tests failing in CI

The e2e tests pass locally but are failing in CI. Usually 1 - 3 tests fail but sometimes we also have that smoke-test failure where no tests run but that can be probably be kept as a separate issue as we have seen it crop up before.

❌ Latest develop where it is still failing https://gitlab.com/gitlab-org/gitter/webapp/pipelines/80349615
❌ Started to fail in https://gitlab.com/gitlab-org/gitter/webapp/pipelines/78214648 (check the retries)
✔ Most recent passing pipeline https://gitlab.com/gitlab-org/gitter/webapp/pipelines/78207589

Doing a diff between the passing commit and the failing commit, I see the backmerge from the 20.0.1 security fix

git diff 0571c2f5 9b053389

The most recent MRs to touch e2e-test.js directly but:

Removes a before hook: https://gitlab.com/gitlab-org/gitter/webapp/merge_requests/1574
- The e2e-tests pass in that MR and after it was merged into develop
Adds a test: https://dev.gitlab.org/gitlab/gitter/webapp/merge_requests/20
- This MR is from after the failures started to happen. And is what spawned me looking into the failures

Dev notes

Running the tests locally

See https://gitlab.com/gitlab-org/gitter/webapp/merge_requests/1585

Successful npm run cypress -- run

$ npm run cypress -- run --env baseUrl=http://localhost:5000,apiBaseUrl=http://localhost:5000/api

> gitter-webapp@19.2.0 cypress C:\Users\MLM\Documents\GitLab\webapp
> cypress "run" "--env" "baseUrl=http://localhost:5000,apiBaseUrl=http://localhost:5000/api"



====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    3.4.0                                                                              │
  │ Browser:    Electron 61 (headless)                                                             │
  │ Specs:      1 found (e2e-test.js)                                                              │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running: e2e-test.js...                                                                  (1 of 1)


  e2e tests
    √ loads homepage (2373ms)
    signed in
      √ shows chat page (1771ms)
      √ can send message (1869ms)
      √ can receive a message (1571ms)
      √ can create a room (2352ms)
      √ can delete account (10183ms)


  6 passing (22s)


  (Results)

  ┌───────────────────────────┐
  │ Tests:        6           │
  │ Passing:      6           │
  │ Failing:      0           │
  │ Pending:      0           │
  │ Skipped:      0           │
  │ Screenshots:  0           │
  │ Video:        true        │
  │ Duration:     22 seconds  │
  │ Spec Ran:     e2e-test.js │
  └───────────────────────────┘


  (Video)

  - Started processing:   Compressing to 32 CRF
  - Finished processing:  C:\Users\MLM\Documents\GitLab\webapp\test\e2e\videos\e2e-test.js.mp4 (3 seconds)


====================================================================================================

  (Run Finished)


      Spec                                                Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ √ e2e-test.js                               00:22        6        6        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    All specs passed!                           00:22        6        6        -        -        -

Progress

Trying to dive into more Cypress and make our tests pass but also give us the information we need in CI to understand why they failed, I did a bunch of stuff in https://gitlab.com/gitlab-org/gitter/webapp/merge_requests/1619

Making tests more robust

In https://gitlab.com/gitlab-org/gitter/webapp/merge_requests/1619, I found some ways to make some tests more robust but they do still fail inexplicably and inconsistently,