curl localhost:3000 from host leads to error: Recv failure Connection reset by peer. Vagrant+Docker

Overview

Stack

I'm running gitlab gdk via vagrant & docker.
I followed the manual to install the vagrant option. During that procedure two changes were needed to have the stack up and running

  1. RubyEncoder needed a new Version
  2. goLang needed a new version (1.14 instead of 1.13)

VirtualBox 6.1.10
The user is member of Dockergroup.
All services are up and running.

Error description

I'm trying to curl localhost:3000 to reach gitlab.

  • When I do that from the vagrant box and docker container, it works
  • When I do that from the ubuntu, it doesnt work - output follows 
    gdkuser@mymachine:~/gitlab-development-kit$ curl localhost:3000 -v --trace -
Warning: --trace overrides an earlier trace/verbose option
== Info: Rebuilt URL to: localhost:3000/
== Info:   Trying 127.0.0.1...
== Info: TCP_NODELAY set
== Info: Connected to localhost (127.0.0.1) port 3000 (#0)
=> Send header, 78 bytes (0x4e)
0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
0010: 48 6f 73 74 3a 20 6c 6f 63 61 6c 68 6f 73 74 3a Host: localhost:
0020: 33 30 30 30 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3000..User-Agent
0030: 3a 20 63 75 72 6c 2f 37 2e 35 38 2e 30 0d 0a 41 : curl/7.58.0..A
0040: 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a       ccept: */*....
== Info: Recv failure: Connection reset by peer
== Info: stopped the pause stream!
== Info: Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Investigation

Port Forwarding looks like that

  • Host (Ubuntu): netstat -tulpn | grep 3000
    tcp6 0 0 :::3000 :::* LISTEN 6508/docker-proxy

  • Vagrant: netstat -tulpn | grep 3000
    tcp 0 0 127.0.0.1:3000 0.0.0.0:* LISTEN 12387/gitlab-workho

  • Docker: netstat -tulpn | grep 3000
    tcp 0 0 127.0.0.1:3000 0.0.0.0:* LISTEN 32478/gitlab-workho

Docker

Docker from my host looks like that:

gdkuser@mymachine:~/gitlab-development-kit$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
549f682a30a4        cb5d370c7aaf        "/bin/sh -c 'supervi…"   About an hour ago   Up About an hour    0.0.0.0:3000->3000/tcp, 127.0.0.1:2222->22/tcp   gitlab-development-kit_default_1592131487

as my host seems to talk with docker-proxy, I've checked tcpdump in docker container to see what's going on there. When I log in docker and tcpdump there, then I see ...

  • for the curl localhost:3000 executed from within docker:

    18:15:22.412071 IP localhost.49248 > localhost.3000: Flags [S], seq 3130406374, win 65495, options [mss 65495,sackOK,TS val 1940365025 ecr 0,nop,wscale 7], length 0
18:15:22.412097 IP localhost.3000 > localhost.49248: Flags [S.], seq 2138251580, ack 3130406375, win 65483, options [mss 65495,sackOK,TS val 1940365025 ecr 1940365025,nop,wscale 7], length 0
18:15:22.412118 IP localhost.49248 > localhost.3000: Flags [.], ack 1, win 512, options [nop,nop,TS val 1940365025 ecr 1940365025], length 0
18:15:22.412198 IP localhost.49248 > localhost.3000: Flags [P.], seq 1:79, ack 1, win 512, options [nop,nop,TS val 1940365025 ecr 1940365025], length 78
18:15:22.412206 IP localhost.3000 > localhost.49248: Flags [.], ack 79, win 511, options [nop,nop,TS val 1940365025 ecr 1940365025], length 0
18:15:22.490506 IP localhost.48377 > localhost.48377: UDP, length 712
18:15:22.516925 IP localhost.54700 > localhost.14268: Flags [P.], seq 11843:13174, ack 973, win 512, options [nop,nop,TS val 1940365129 ecr 1940354813], length 1331
18:15:22.517066 IP localhost.3000 > localhost.49248: Flags [P.], seq 1:941, ack 79, win 512, options [nop,nop,TS val 1940365130 ecr 1940365025], length 940
18:15:22.517091 IP localhost.49248 > localhost.3000: Flags [.], ack 941, win 505, options [nop,nop,TS val 1940365130 ecr 1940365130], length 0
18:15:22.517128 IP localhost.14268 > localhost.54700: Flags [P.], seq 973:1054, ack 13174, win 512, options [nop,nop,TS val 1940365130 ecr 1940365129], length 81
18:15:22.517137 IP localhost.54700 > localhost.14268: Flags [.], ack 1054, win 512, options [nop,nop,TS val 1940365130 ecr 1940365130], length 0
18:15:22.517296 IP localhost.49248 > localhost.3000: Flags [F.], seq 79, ack 941, win 512, options [nop,nop,TS val 1940365130 ecr 1940365130], length 0
18:15:22.517458 IP localhost.3000 > localhost.49248: Flags [F.], seq 941, ack 80, win 512, options [nop,nop,TS val 1940365130 ecr 1940365130], length 0
18:15:22.517473 IP localhost.54700 > localhost.14268: Flags [P.], seq 13174:13817, ack 1054, win 512, options [nop,nop,TS val 1940365130 ecr 1940365130], length 643
18:15:22.517481 IP localhost.49248 > localhost.3000: Flags [.], ack 942, win 512, options [nop,nop,TS val 1940365130 ecr 1940365130], length 0

  • for the curl localhost:3000 executed from host / ubuntu:

    17:55:24.866087 IP 172.17.0.1.55052 > 89e4c7aa5aad.3000: Flags [S], seq 1766463375, win 64240, options [mss 1460,sackOK,TS val 2894194925 ecr 0,nop,wscale 7], length 0
17:55:24.866139 IP 89e4c7aa5aad.3000 > 172.17.0.1.55052: Flags [R.], seq 0, ack 1766463376, win 0, length 0

Gitlab Logs

For requests originating from ubuntu I don't see any logs, for requests from vagrant I see traffic in /home/vagrant/gdk/gitlab/log# vi development.log when I execute curl localhost:3000

Started GET "/" for 127.0.0.1 at 2020-06-15 20:21:32 +0000
  ^[[1m^[[36mApplicationSetting Load (3.6ms)^[[0m  ^[[1m^[[34mSELECT "application_settings".* FROM "application_settings" ORDER BY "application_settings"."id" DESC LIMIT $1^[[0m  [["LIMIT", 1]]
  ▒~F▒ app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'
Processing by RootController#index as */*
Redirected to http://localhost:3000/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 7ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 10147)

Thus the request seems to arrive within docker, but doesn't even arrive at Gitlab's Log system If the request is being sent from within docker, then it's processed as expected.

Is there a way to enable more details in Gitlab Logging? How could I check the first touchpoint with Gitlab, to narrow down the topic a little bit more?

Environment

  • Operating System: Ubuntu 18.04 LTS
  • Ruby version: ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
  • GDK version: 72717dd
Edited by 🤖 GitLab Bot 🤖