Cypress integration test using docker-compose passes locally but fails in GitLab CI with docker:dind
Summary
A job in my pipeline uses docker-compose with Cypress to run integration tests. The docker-compose.ci.yml file contains the following services:
postgres-
backend# django webserver -
asgiserver# django channels daphne server -
nginx# serves Vue application that makes websocket requests redis
These containers are built and run as described in the integration-tests.sh (see below) and then start the cypress service with docker-compose which runs integration tests.
Out of 5 Cypress integration test, 4 passes locally. The only test that is failing tests websocket-related behavior.
Here is the repo: https://gitlab.com/verbose-equals-true/django-postgres-vue-gitlab-ecs
Steps to reproduce
To reproduce this locally, you can run ./integration-tests.sh (the same script that I'm using in the failing gitlab-ci.yml job). This will build and run containers with docker-compose:
echo "Starting services"
docker-compose -f docker-compose.ci.yml up -d --build
sleep 20
echo "Services are up and ready"
docker-compose -f docker-compose.ci.yml -f cypress.yml up --exit-code-from cypress
echo "Cypress tests passed successfully."
echo "Stopping docker compose..."
docker-compose -f docker-compose.ci.yml -f cypress.yml down.gitlab-ci.yml
e2e cypress tests with docker-compose:
stage: integration
image: docker:stable
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
services:
- docker:dind
before_script:
- apk add --update py-pip
- pip install docker-compose~=1.23.0
script:
- ls
- pwd
- sh integration-tests.sh
after_script:
- echo "Integration tests passed."
artifacts:
paths:
- cypress/videos/
- tests/screenshots/
expire_in: 7 daysActual behavior
A cypress test that tests websocket connection behavior fails in GitLab CI. The actual pipeline is passing, but that is only so that the job can capture artifacts (Cypress records video of each test).
The 5 cypress tests cover the following:
-
test_api.js: testscy.requestcan reach an API endpoint that returns a static response -
test_homepage.js: tests that cypress can access the Vue application -
test_login.js: tests login functionality (tests the connection to thepostgresdatabase container) -
test_logout.js: tests a user can logout; uses customloginCommand instead of login manually as intest_login.js -
test_ws_connection.js: presses a button that sends a websocket message and tests that the websocket response from the server triggers the creation of a notification element in the DOM.
In the GitLab CI logs, I can see that the websocket connection is successful:
asgiserver | 172.19.0.6:60788 - - [29/Jul/2019:03:46:26] "WSCONNECTING /ws/ping-pong/" - -
asgiserver | specific.OmbkseAH!hVmbYwSYgRiZ
asgiserver | 172.19.0.6:60788 - - [29/Jul/2019:03:46:27] "WSCONNECT /ws/ping-pong/" - -In the video artifacts that are linked to in the job log, I see that the PING button is clicked, but I don't see the expected notification displayed in the UI.
Expected behavior
All Cypress tests running in docker-compose should pass in GitLab CI as they pass when running locally on docker-compose. When I run docker-compose locally, I see the following logs:
asgiserver | 192.168.160.6:42158 - - [29/Jul/2019:04:10:29] "WSCONNECTING /ws/ping-pong/" - -
asgiserver | specific.cHSRkyiZ!lrvzSAdPrLsn
asgiserver | 192.168.160.6:42158 - - [29/Jul/2019:04:10:29] "WSCONNECT /ws/ping-pong/" - -
asgiserver | received message
asgiserver | sending ws response
✓ User sees PONG message from websocket PING message (721ms)
cypress | Relevant logs and/or screenshots
Here are logs from the failing job
https://gitlab.com/verbose-equals-true/django-postgres-vue-gitlab-ecs/-/jobs/260939805
Some things to note:
- A websocket connection seems to be made (
WSCONNECTshows in the logs ofasgiserver), but print statements that are run when a new message is received do show in the logs as they do when the tests are run locally. - nginx routes
/api/traffic tobackend:8000, and routes/ws/traffic toasgiserver:9000
job log
cypress | Running: test_ws_connection.js... (5 of 5)
cypress |
cypress |
cypress | Test websockets
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "POST /api/auth/obtain_token/ HTTP/1.1" 200 438 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /examples/websockets HTTP/1.1" 200 2405 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /css/app.32650074.css HTTP/1.1" 200 268424 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/app.0f31fa3b.js HTTP/1.1" 200 18867 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /css/701a639c.1c285fc2.css HTTP/1.1" 200 233 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/2d0abad9.36e252bb.js HTTP/1.1" 200 323 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/2d0b66f7.b76d307a.js HTTP/1.1" 200 416 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/vendor.b46ce879.js HTTP/1.1" 200 629431 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/2d0da96f.c6047424.js HTTP/1.1" 200 1048 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/2d0dd014.daa8bd7a.js HTTP/1.1" 200 2228 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/2d0e8be2.feb6f7cc.js HTTP/1.1" 200 356 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/2d207d33.f6123fc4.js HTTP/1.1" 200 500 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/2d22d072.f005affc.js HTTP/1.1" 200 795 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/4b47640d.83b94a0a.js HTTP/1.1" 200 12364 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:55 +0000] "GET /js/701a639c.77289ef5.js HTTP/1.1" 200 639 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:56 +0000] "GET /api/users/profile/ HTTP/1.1" 200 72 "http://nginx/examples/websockets" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:56 +0000] "GET /fonts/flUhRq6tzZclQEJ-Vdg-IuiaDsNcIhQ8tQ.0509ab09.woff2 HTTP/1.1" 200 60840 "http://nginx/css/app.32650074.css" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:56 +0000] "GET /fonts/KFOmCnqEu92Fr1Mu4mxM.60fa3c06.woff HTTP/1.1" 200 20268 "http://nginx/css/app.32650074.css" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
nginx | 172.19.0.7 - - [29/Jul/2019:03:00:56 +0000] "GET /fonts/KFOlCnqEu92Fr1MmEU9fBBc-.87284894.woff HTTP/1.1" 200 20464 "http://nginx/css/app.32650074.css" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
asgiserver | 172.19.0.6:42468 - - [29/Jul/2019:03:00:58] "WSCONNECTING /ws/ping-pong/" - -
asgiserver | specific.BkmJceJr!bNkFMYoYNGPI
asgiserver | 172.19.0.6:42468 - - [29/Jul/2019:03:00:59] "WSCONNECT /ws/ping-pong/" - -
cypress |
1) User sees PONG message from websocket PING message
cypress |
cypress |
cypress | 0 passing (6s)
cypress | 1 failing
cypress |
cypress | 1) Test websockets User sees PONG message from websocket PING message:
cypress | CypressError: Timed out retrying: Expected to find element: '.pong', but never found it.
cypress | at Object.cypressErr (http://nginx/__cypress/runner/cypress_runner.js:65727:11)
cypress | at Object.throwErr (http://nginx/__cypress/runner/cypress_runner.js:65692:18)
cypress | at Object.throwErrByPath (http://nginx/__cypress/runner/cypress_runner.js:65719:17)
cypress | at retry (http://nginx/__cypress/runner/cypress_runner.js:59237:16)
cypress | at http://nginx/__cypress/runner/cypress_runner.js:51312:18
cypress | at tryCatcher (http://nginx/__cypress/runner/cypress_runner.js:131273:23)
cypress | at Promise._settlePromiseFromHandler (http://nginx/__cypress/runner/cypress_runner.js:129291:31)
cypress | at Promise._settlePromise (http://nginx/__cypress/runner/cypress_runner.js:129348:18)
cypress | at Promise._settlePromise0 (http://nginx/__cypress/runner/cypress_runner.js:129393:10)
cypress | at Promise._settlePromises (http://nginx/__cypress/runner/cypress_runner.js:129468:18)
cypress | at Async._drainQueue (http://nginx/__cypress/runner/cypress_runner.js:126197:16)
cypress | at Async._drainQueues (http://nginx/__cypress/runner/cypress_runner.js:126207:10)
cypress | at Async.drainQueues (http://nginx/__cypress/runner/cypress_runner.js:126081:14)
cypress | at <anonymous>
cypress |
cypress |
cypress |
cypress |
cypress | (Results)
cypress |
cypress | ┌─────────────────────────────────────┐
cypress | │ Tests: 1 │
cypress | │ Passing: 0 │
cypress | │ Failing: 1 │
cypress | │ Pending: 0 │
cypress | │ Skipped: 0 │
cypress | │ Screenshots: 1 │
cypress | │ Video: true │
cypress | │ Duration: 6 seconds │
cypress | │ Spec Ran: test_ws_connection.js │
cypress | └─────────────────────────────────────┘
cypress |
cypress |
cypress | (Screenshots)
cypress |
cypress | - /e2e/cypress/screenshots/test_ws_connection.js/Test websockets -- User sees PONG message from websocket PING message (failed).png (1280x720)
cypress |
nginx | 172.19.0.7 - - [29/Jul/2019:03:01:01 +0000] "GET /ws/ping-pong/ HTTP/1.1" 101 4 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.2.0 Chrome/59.0.3071.115 Electron/1.8.2 Safari/537.36"
asgiserver | 172.19.0.6:42468 - - [29/Jul/2019:03:01:01] "WSDISCONNECT /ws/ping-pong/" - -
cypress |
cypress | (Video)
cypress |
cypress | - Started processing: Compressing to 32 CRF
cypress | - Finished processing: /e2e/cypress/videos/test_ws_connection.js.mp4 (3 seconds)
cypress |
cypress |
cypress | ====================================================================================================
cypress |
cypress | (Run Finished)
cypress |
cypress |
cypress | Spec Tests Passing Failing Pending Skipped
cypress | ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
cypress | │ ✔ test_api.js 00:02 1 1 - - - │
cypress | ├────────────────────────────────────────────────────────────────────────────────────────────────┤
cypress | │ ✔ test_homepage.js 00:01 1 1 - - - │
cypress | ├────────────────────────────────────────────────────────────────────────────────────────────────┤
cypress | │ ✔ test_login.js 00:04 1 1 - - - │
cypress | ├────────────────────────────────────────────────────────────────────────────────────────────────┤
cypress | │ ✔ test_logout.js 00:03 1 1 - - - │
cypress | ├────────────────────────────────────────────────────────────────────────────────────────────────┤
cypress | │ ✖ test_ws_connection.js 00:06 1 - 1 - - │
cypress | └────────────────────────────────────────────────────────────────────────────────────────────────┘
cypress | 1 of 5 failed (20%) 00:17 5 4 1 - - Environment description
I am running this on regular shared runners on GitLab.com. I am unable to run this job with my locally registered gitlab-runner (I am able to run other non-docker jobs). I have tried using the --docker-privileged flag, but this did not work. This isn't the main issue, but I would prefer to debug this issue locally rather than use shared GitLab.com runners. It would be helpful to know what the outcome of this job is using gitlab-runner locally.
I have tried to set up my local gitlab runner by editing /etc/gitlab-runner/config.toml, but this didn't work.
config.toml contents
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Verbose Equals True Docker Runner"
url = "https://gitlab.com"
token = "<removed>"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.custom]
run_exec = ""Used GitLab Runner version
Version: 12.1.0
Git revision: de7731dd
Git branch: 12-1-stable
GO version: go1.8.7
Built: 2019-07-19T13:53:04+0000
OS/Arch: linux/amd64Possible fixes
To further diagnose, it would be helpful to get console or network errors messages from the cypress test that when it fails in CI.
I should also write a test that tests redis connectivity, since I don't have any other guarantee that it is working.
Edited by Brian Caffey