Auto DevOps support for running browser tests (eg. selenium)
Problem to solve
Right now the tests in Auto DevOps are running in herokuish test image but we don't have any way to control the image being run in so that we have access to chromedriver etc. needed for running these kinds of tests.
It's also very common to have these kinds of tests and you really want your CI environment to also be able to run them.
Further details
(Include use cases, benefits, and/or goals)
Proposal
Option 1: Hooks: (script committed to repo)
A specific solution of having a job run that is already set up with chrome headless for you and simply runs a predefined script in your repo if it exists like ci/browser-tests
but still you will probably end up needing to install a bunch of dependency for your programming language. The major disadvantage here is that ci/browser-tests
would need to include a bunch of commands for installing all the dependencies you might need (eg. ruby or java or whatever else) simply because we don't know what docker image to use for your job.
Option 2: Support for specific languages/frameworks
Let's say we want to support running rspec features/
for your browser tests. Then we could add a job to the auto devops template that looks something like:
rspec_features:
image: image_with_ruby_and_chrome_headless
script:
- bundle install --without development --without production
- bundle exec rspec features/
only:
variables:
- $RSPEC_FEATURE_TESTS
This way if somebody matches this standard all they need to do is set $RSPEC_FEATURE_TESTS
in their env settings and all off a sudden we're running their feature tests in the right docker image for them. This obviously has the disadvantage that we'd need to implement this for every type of browser testing we want to support.
Option 3: Hooks: (CI YML committed to repo)
There are many ways to tackle this specific problem but maybe we should investigate some more generic solutions proposed that could allow people to do other kinds of tests that are difficult with herokuish right now:
- https://gitlab.com/gitlab-org/gitlab-ce/issues/52679
- https://gitlab.com/gitlab-org/gitlab-ce/issues/47234
The more generic solutions may fix this problem while also allowing a bunch of different types of steps to be added to your pipeline that compliment Auto DevOps.
What does success look like, and how can we measure that?
We can run capybara or other similar selenium based tests without forking the Auto DevOps template.