Job services cannot communicate with each other on GitLab.com SaaS Runners on Linux
Summary
A integration testing job from the CI pipeline is configured to use two services - a database, and a web-app image (built on a previous stage). The web app fails to connect to the DB.
They are both accessible from the job's main script, and the DB URL used there is the same as the one from inside the web-app's image.
Steps to reproduce
It happens consistently on the GitLab.com's CI pipeline, running on shared runners.
Example Project
The project is nodejs+express
web app, using mongodb
database. The setup is quite straightforward, using popular base images, with the mongodb
, actually not creating a custom one at all.
The .gitlab-ci.yml
file gives quite nice overview:
stages:
- build
- test
workflow:
rules:
- if: $CI_MERGE_REQUEST_ID # Execute jobs in merge request context
- if: $CI_COMMIT_BRANCH == 'master' # Execute jobs when a new commit is pushed to master branch
- if: $CI_COMMIT_BRANCH == 'dev' # Execute jobs when a new commit is pushed to dev branch
variables:
APP_IMAGE: my-webapp
DB_IMAGE: my-testdb
APP_CONTAINER: $CI_REGISTRY_IMAGE/$APP_IMAGE:$CI_COMMIT_REF_SLUG
build-app:
stage: build
image: docker:19.03.1
services:
- docker:19.03.1-dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build -t "$APP_CONTAINER" src
- docker push "$APP_CONTAINER"
test-app:
stage: test
image: node:latest
services:
- name: $APP_CONTAINER
alias: $APP_IMAGE
- name: mongo:4.2-bionic
variables:
SERVICE_URL: "http://$CI_REGISTRY-my-system-backend-$APP_IMAGE:5000/"
MONGODB_URL: "mongodb://mongo:27017/MainDb"
script:
- npm install
- npm run gen-fake
- npm run verify
Note: The full APP_CONTAINER
name is used, because it turns out the short-name fails to be recognized, but that's another - minor - bug.
What is the current bug behavior?
The service running from the APP_CONTAINER
fails to connect to the service running from mongo
container, hanging the job for quite a long time.
Both are accessible from the job's main script, though.
What is the expected correct behavior?
The service running from APP_CONTAINER
image to be able to connect to the mongo
service directly.
Relevant logs and/or screenshots
Here's the job log (before_script
and script
):
npm install
... (success) ...
npm run gen-fake
> my-backend@0.1.0 gen-fake /builds/my/my-backend
> node specs/db/GenerateFake.js
Connecting to: mongodb://mongo:27017/MainDb
Db connected!
5 documents added to 'A'.
2 documents added to 'B'.
2 documents added to 'C'.
5 documents added to 'D'.
3 documents added to 'E'.
5 documents added to 'F'.
Disconnecting from Db...
Disconnected!
npm run verify
> my-backend@0.1.0 verify /builds/my/my-backend
> node tests/Demo.test.js
Server URL to be used: 'http://registry.gitlab.com-my-my-backend-my-webapp:5000/'
MyApp Demo Express is ON
{
"Db": {
"useNewUrlParser": true,
"useCreateIndex": true,
"useFindAndModify": false,
"useUnifiedTopology": true,
"url": "mongodb://mongo:27017/MainDb"
}
}
The npm run gen-fake
steps connects to the DB.
The npm run verify
's report message MyApp Demo Express is ON
shows that the app is connected as well, and the JSON afterwards shows the configuration used - the url
is the correct one.
But the app itself, fails to connect to DB - the job hangs, and checking the logs from inside the container shows that.
Output of checks
This bug happens on GitLab.com
Used are shared runners.
Possible fixes
I've seen several issues on similar/same problems. However, all of them are quite old and closed, which makes me think, this is an already solved problem, reoccurring again.