Experiment with Cypress for smoke testing

Problem

Currently, we use Selenium to run end-to-end tests for the GitLab application, and this brings lots of issues. I'll list some of them below:

• Selenium drives the browser through remote commands via the network, and this can cause test failures due to network issues.
• We need to install lots of different libraries to make testing works, such as Capybara, RSpec, and others.
• Developers usually find it challenging to write end-to-end tests with Selenium-based frameworks, due to unnecessary complexities, such as the use of too many levels of abstractions, and especially, due to tests flakiness.

Proposal to solve the problem

I propose that we experiment with Cypress for our smoke test suite to evaluate if it can provide us with more stable tests, but also, as a way to get more contributions in terms of end-to-end testing from engineers outside of the Quality department.

Pros and cons

Below I list some pros and cons of using Cypress as a test automation framework.

Pros

• Cypress is open source
• Easy to set up and start using. When you open Cypress in interactive mode for the first time, it already bootstraps the files and directories structure needed to start using it
• All-in-one testing framework, assertion library, with mocking and stubbing, all without Selenium
• Cypress focuses on doing end-to-end testing really well - it specializes in one thing - creating a great experience while you write end-to-end tests for web applications
• Cypress works on any front-end framework or website
• Cypress is for developers and test automation engineers, and it makes test-driven development a reality for end-to-end testing
• Cypress runs much faster due to running in the same run loop as the application(more about it in a next item)
• Front-end engineers working with Vue.js would be able to contribute with end-to-end tests since Cypress tests are written in modern JavaScript, which is the language they are used to when writing Vue apps
• Cypress is executed in the same run loop as the application. Having access to both parts (front and back) gives us the ability to respond to application's events in real-time, while at the same time work outside of the browser for tasks that require a higher privilege. It also allows us to intercept requests, change the application state to create pre-conditions for the tests, simulate race conditions, and more
• Cypress also operates at the network layer by reading and altering web traffic on the fly. This enables Cypress not only to modify everything coming in and out of the browser but also to change the code that may interfere with its ability to automate the browser
• Cypress is capable of delivering more consistent results
• Because Cypress is installed locally, it can additionally tap into the operating system itself for automation tasks. This makes performing tasks such as taking screenshots, recording videos, general file system operations and network operations possible
• Cypress automatically waits for the application to reach the desired state before moving on. It automatically waits for elements to exist and will never yield you stale elements that have been detached from the DOM
• Cypress even knows how fast an element is animating and will wait for it to stop! Additionally, it automatically waits for elements to become visible, to become enabled, and to stop being covered. When pages begin to transition, Cypress will pause command execution until the following page is fully loaded. You can even tell Cypress to wait on specific network requests to finish
• On Cypress debuggability is first-class - above all else, Cypress has been built for usability. There are hundreds of custom error messages describing the exact reason Cypress failed your test
• Cypress takes snapshots of your application and enables you to time travel back to the state it was in when commands ran
• Cypress has ​excellent documentation, where they not only show how to use the tool, but also how to implement best practices, and especially important, how to avoid bad practices
• No need for Page Objects. On Cypress the idea is that tests should be as isolated as possible, not requiring an extra level of abstraction such as Page Objects
• Cypress provides us with the ability to create custom commands for reusability purposes
• Cypress also provides us with the ability to create test fixtures that can be reused by tests
• Cypress has now cross-browser support (Chrome, Firefox, and Edge.)

Cons

• No support for multiple browser tabs
• No support to drive two browsers at the same time
• Testing file uploads is application-specific
• iframe support is somewhat limited

Even having these trade-offs, Cypress explains how having many of these boundaries are actually good to have.

Sample project

I have created a sample project that runs the following tests using Cypress:

• Assign an issue
• Close an issue
• Close an issue using quick actions
• Reopen a closed issue
• Comment on an issue
• Create an access token
• Create a group
• Create a sub-group
• Create a group label
• Create an issue
• Create and commit a file
• Create a project
• Create a project milestone
• Visualize opened and closed issues on issue boards
• Set a milestone to an issue
• Set a label to an issue
• Login
• Logout

Tests' pre-conditions are created by calling the public API, which makes tests faster, and more reliable.

Demo

I have presented a demo of this sample project (when it was not so mature as it currently is), and you can watch it in this video. To go directly to the demo, fast-forward to 21:30.

Cc @gl-quality.