Trial running GDK in docker to quickly run E2E tests in MRs
Closing summary
Although it was possible to run GDK in a container and run the tests against it successfully on my machine, even the relatively higher capacity n2d-standard-8
machines we use in the GitLab QA project weren't sufficient to run the tests reliably on the same machine as the GDK container.
One alternative would be to increase the runner resources until the tests do run reliably.
Another alternative is to deploy GDK as a temporary test environment in each MR (similar to Review Apps). Some advantages of this approach compared to the previous include:
- Faster updates and startup after the initial deployment.
- The option to run tests from multiple jobs in parallel against the same environment.
- The ability to inspect the environment itself during/after test runes.
That alternative will be explored in https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1285
Original issue follows
Rationale
When we run E2E tests in CI currently, we build and install GitLab before running the tests. This can add an hour or more of a delay before test results are available.
It would take very significant improvements to the build and install processes to reduce that time to a reasonable amount.
In contrast, we use GDK in development and it takes only a few minutes to update and start. If we could use GDK in CI then we could run E2E tests in MRs and get feedback to engineers much much sooner.
Proposal
Define a QA GDK Dockerfile that builds upon the GDK Dockerfile that is used to test GDK itself.
The new QA GDK Dockerfile will allow the code from an MR to be checked out into GDK running in a docker container, and then an instance of GitLab will be started that tests can be run against.
-
Add qa/Dockerfile
togitlab-development-kit
or togitlab-org/gitlab
to build a GDK test image -
Add a job to https://gitlab.com/gitlab-org/gitlab that will launch a GitLab instance using the image above, and using the GitLab source code from an MR, and then run the E2E tests against it. -
Build the image and keep it up-to-date automatically in https://gitlab.com/gitlab-org/gitlab-build-images/ -
TDB
Caveats
GDK is a development environment, so it's not a substitute for running E2E tests in a production-like environment. Therefore, even if we run tests on GDK in MRs, we should continue to run the tests against omnibus-gitlab in master.
We currently also run tests against Review Apps. We could continue to do that, but if tests against GDK are sufficient we might choose to allow tests to be run against Review Apps on demand (e.g., manually, or automatically but only if a label is present, i.e., pipeline:run-review-app)