Gitlab runner dind service cannot be used with Testcontainers

Summary

A lot of Java projects are using the Testcontainers.org library to run services during unit testing. One such use case is starting a database service for each unit test, allowing you to run each test in a clean environment that can't conflict with tests running before, after or concurrently.

When using Gitlab runner with the docker executor on a recent Docker Engine version - where links are deprecated and the default bridge network isolates containers - then the docker:dind service cannot be used to run Testconainers based tests: either the services started cannot be accessed due to new network isolation feature of the default bridge network, or - when running on a user network - the dind service cannot be accessed at all.

Steps to reproduce

General setup

Ubuntu 16.04
Gitlab runner 10.6.0 is installed from the Gitlab runner linux repository.
Docker Engine 18.03.1 installed from docker.com repository
Gitlab runner is configured with the docker executor, "privileged" mode enabled and default networking.

.gitlab-ci.yml setup:

add docker:dind under the services section
add DOCKER_HOST=tcp://docker:2375 under the variables section

This CI configuration works well on gitlab.com and so it is the preferred method when using Gitlab runner as a "specific runner" for a gitlab.com project.

Option 1 - use the default configuration:

Actual behavior

Testconainers docker client starts the test container under the dind service, then when the test tries to access the container service port mapped from the internal container to the dinds service container, it is stopped by Docker's network isolation feature and the test gets a timeout on the connection attempt:

Example log output:

[main] INFO 🐳 [mariadb:latest] - Creating container for image: mariadb:latest
[main] INFO 🐳 [mariadb:latest] - Starting container with ID: dc852d8ef6542af1ac62f9da3ae0885cd0dfd73781f7b14267ec126f4d5a77c4
[main] INFO 🐳 [mariadb:latest] - Container mariadb:latest is starting: dc852d8ef6542af1ac62f9da3ae0885cd0dfd73781f7b14267ec126f4d5a77c4
[main] INFO 🐳 [mariadb:latest] - Waiting for database connection to become available at jdbc:mysql://127.0.0.1:32770/a2billing?useSSL=false&allowMultiQueries=true using query 'SELECT 1'
[testcontainers-ryuk] WARN org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at 172.18.0.1:32768
java.net.ConnectException: Connection timed out (Connection timed out)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:118)
	at java.lang.Thread.run(Thread.java:748)
[main] ERROR 🐳 [mariadb:latest] - Could not start container
org.rnorth.ducttape.TimeoutException: org.rnorth.ducttape.TimeoutException: java.util.concurrent.TimeoutException
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:53)
	at org.testcontainers.containers.JdbcDatabaseContainer.waitUntilContainerStarted(JdbcDatabaseContainer.java:94)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:258)
	at org.testcontainers.containers.GenericContainer.lambda$start$0(GenericContainer.java:209)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:207)
	at io.kaller.backend.helpers.MockDatabaseRule$1.evaluate(MockDatabaseRule.java:199)
	at io.vertx.ext.unit.junit.RunTestOnContext$1.evaluate(RunTestOnContext.java:93)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Caused by: org.rnorth.ducttape.TimeoutException: java.util.concurrent.TimeoutException
	at org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:59)
	at org.rnorth.ducttape.timeouts.Timeouts.getWithTimeout(Timeouts.java:32)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:38)
	... 16 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:54)
	... 18 more

In the log above we can see that the "Ryuk" container started by Testcontainers to perform end-of-test cleanups, which is the first container started, cannot be contacted on the mapped port 32768 (the first port mapping allocated by the docker server in the dind container) due to networking isolation. The Testconainers library tries a few times and then throws a timeout error.

Expected behavior

Testcontainers should be able to start containers with mapped ports and connecting to them without networking issues.

Here is a log example from a different project (using a similar setup) that runs on gitlab.com's shared runners:

[Fri Apr 13 21:42:45 UTC 2018] INFO: org.testcontainers.containers.GenericContainer tryStart - Creating container for image: mariadb:latest 
[Fri Apr 13 21:42:45 UTC 2018] INFO: org.testcontainers.containers.GenericContainer tryStart - Starting container with ID: 21b28de75667cbafb697c48ba925bbd580183ac6c6ea84085fdd60414e663f70 
[Fri Apr 13 21:42:45 UTC 2018] INFO: org.testcontainers.containers.GenericContainer tryStart - Container mariadb:latest is starting: 21b28de75667cbafb697c48ba925bbd580183ac6c6ea84085fdd60414e663f70 
[Fri Apr 13 21:42:46 UTC 2018] INFO: org.testcontainers.containers.JdbcDatabaseContainer waitUntilContainerStarted - Waiting for database connection to become available at jdbc:mysql://test:test@172.18.0.1:32770/database?useSSL=false&allowMultiQueries=true using query 'SELECT 1' 
[Fri Apr 13 21:43:01 UTC 2018] INFO: org.testcontainers.containers.JdbcDatabaseContainer lambda$waitUntilContainerStarted$0 - Obtained a connection to container (jdbc:mysql://test:test@172.18.0.1:32770/database?useSSL=false&allowMultiQueries=true)

Option 2 - create a user network for the Gitlab runner

Setup:

Create a user network: `docker network create gitlab-runner-net --subnet 172.30.0.0/16'
Edit Gitlab runner config.toml file and add network_mode = "gitlab-runner-net" under [runners.docker]

Actual behavior

Hostname resolution on user networks is not using the --link feature aliases and so the Testcontainers docker client can't find the dind service container's IP address.

Example log output:

[main] ERROR org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - ping failed with configuration Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://docker:2375
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://docker:2375,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfig=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION}]'
 due to org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:51)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy.ping(DockerClientProviderStrategy.java:175)
	at org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy.test(EnvironmentAndSystemPropertyClientProviderStrategy.java:42)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$2(DockerClientProviderStrategy.java:111)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
	at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:302)
	at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:731)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:469)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:146)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:99)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:142)
	at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:37)
	at org.testcontainers.containers.MariaDBContainer.<init>(MariaDBContainer.java:19)
	at io.kaller.backend.helpers.MariaDBTestContainer.<init>(MariaDBTestContainer.java:17)
	at io.kaller.backend.helpers.MockDatabaseRule.<init>(MockDatabaseRule.java:42)
	at io.kaller.backend.service.OutboundTest.<clinit>(OutboundTest.java:29)
	at sun.misc.Unsafe.ensureClassInitialized(Native Method)
	at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
	at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
	at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
	at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
	at java.lang.reflect.Field.get(Field.java:393)
	at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
	at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
	at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
	at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
	at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Caused by: java.net.UnknownHostException: docker
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at java.net.InetAddress.getByName(InetAddress.java:1076)
	at org.testcontainers.shaded.io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146)
	at org.testcontainers.shaded.io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.testcontainers.shaded.io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143)
	at org.testcontainers.shaded.io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43)
	at org.testcontainers.shaded.io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63)
	at org.testcontainers.shaded.io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55)
	at org.testcontainers.shaded.io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57)
	at org.testcontainers.shaded.io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32)

Used GitLab Runner version

# gitlab-runner --version
Version:      10.6.0
Git revision: a3543a27
Git branch:   10-6-stable
GO version:   go1.9.4
Built:        2018-03-22T08:34:11+00:00
OS/Arch:      linux/amd64

Edited Apr 30, 2018 by Oded Arbel