Skip to content

Too many open files error when running on macOS in Docker

Summary

When running GitLab in docker on macOS, GitLabs raises multiple errors due to too many open files. Not possible to push to repositories. Nothing is possible only restart solve problem.

503
Whoops, GitLab is currently unavailable.

Docker process com.docker.osxfx open files reaches limit 10245 and then error appears.

Docker Desktop (macOS): Version 2.0.0.3 (31259) macOS: 10.13.6

Restarting GitLab container results in immediate lowering opened files by com.docker.osxfx -> Docker correctly release file descriptors

Steps to reproduce

  1. Install Docker Desktop 2.0.0.3 on macOS
  2. docker-compose up -d with docker-compose.yml
  3. Work with GitLab
version: '2'
services:
  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    container_name: gitlab-latest
    hostname: imac.local
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://imac.local:9080'
        gitlab_rails['gitlab_shell_ssh_port'] = 9022
    ports:
      - '9443:443' # https port
      - '9080:9080' # http port
      - '9022:22' # docker:host
    volumes:
      - '~/Docker/GitLab/config:/etc/gitlab:cached'
      - '~/Docker/GitLab/logs:/var/log/gitlab:cached'
      - '~/Docker/GitLab/data:/var/opt/gitlab:cached'

What is the current bug behavior?

Open files per com.docker.osxfs grows up to 10245 when Postgresql starts to complain about too many open files

PID=`pgrep com.docker.osxfs`
lsof -n -p $PID | wc

What is the expected correct behavior?

Opened files per process should not reach more then 2000

Relevant logs

Relevant logs 
Completed 200 OK in 55ms (ActiveRecord: 7.1ms)
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/project/avatar/19/home.png (0.8ms)
Completed 200 OK in 65ms (ActiveRecord: 7.5ms)
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/project/avatar/8/icon.png (0.9ms)
Completed 200 OK in 41ms (ActiveRecord: 6.0ms)
Started GET "/inventloop/other" for 172.18.0.1 at 2019-05-07 16:44:10 +0000
Processing by GroupsController#show as HTML
Parameters: {"id"=>"inventloop/other"}
Completed 200 OK in 148ms (Views: 91.9ms | ActiveRecord: 26.5ms)
Started GET "/groups/inventloop/other/-/children.json" for 172.18.0.1 at 2019-05-07 16:44:10 +0000
Processing by Groups::ChildrenController#index as JSON
Parameters: {"group_id"=>"inventloop/other"}
Started GET "/uploads/-/system/appearance/header_logo/1/inventloop-logo-tiny_2x.png" for 172.18.0.1 at 2019-05-07 16:44:10 +0000
Started GET "/uploads/-/system/group/avatar/3/inventloop.png?width=15" for 172.18.0.1 at 2019-05-07 16:44:10 +0000
Started GET "/uploads/-/system/user/avatar/2/avatar.png?width=23" for 172.18.0.1 at 2019-05-07 16:44:10 +0000
Processing by UploadsController#show as HTML
Parameters: {"model"=>"appearance", "mounted_as"=>"header_logo", "id"=>"1", "filename"=>"inventloop-logo-tiny_2x.png"}
Processing by UploadsController#show as HTML
Processing by UploadsController#show as HTML
Parameters: {"width"=>"15", "model"=>"group", "mounted_as"=>"avatar", "id"=>"3", "filename"=>"inventloop.png"}
Parameters: {"width"=>"23", "model"=>"user", "mounted_as"=>"avatar", "id"=>"2", "filename"=>"avatar.png"}
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/group/avatar/3/inventloop.png (1.2ms)
Completed 200 OK in 33ms (ActiveRecord: 4.8ms)
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/appearance/header_logo/1/inventloop-logo-tiny_2x.png (0.7ms)
Completed 200 OK in 54ms (ActiveRecord: 30.2ms)
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/user/avatar/2/avatar.png (2.7ms)
Completed 200 OK in 49ms (ActiveRecord: 8.4ms)
Completed 200 OK in 579ms (Views: 0.5ms | ActiveRecord: 38.8ms)
Started GET "/-/metrics" for 127.0.0.1 at 2019-05-07 16:44:14 +0000
Processing by MetricsController#index as HTML
Completed 200 OK in 30ms (Views: 0.6ms | ActiveRecord: 0.0ms)
Started GET "/fbucek/invsdk.git/info/refs?service=git-upload-pack" for 172.18.0.1 at 2019-05-07 16:44:27 +0000
Started GET "/fbucek/fileork.git/info/refs?service=git-upload-pack" for 172.18.0.1 at 2019-05-07 16:44:27 +0000
Processing by Projects::GitHttpController#info_refs as /
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"fbucek", "project_id"=>"invsdk.git"}
Processing by Projects::GitHttpController#info_refs as /
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"fbucek", "project_id"=>"fileork.git"}
Completed 200 OK in 29ms (Views: 1.1ms | ActiveRecord: 3.6ms)
Completed 200 OK in 39ms (Views: 0.6ms | ActiveRecord: 4.2ms)
Started GET "/fbucek/code-management.git/info/refs?service=git-upload-pack" for 172.18.0.1 at 2019-05-07 16:44:27 +0000
Started GET "/jobs/job-tasks.git/info/refs?service=git-upload-pack" for 172.18.0.1 at 2019-05-07 16:44:27 +0000
Processing by Projects::GitHttpController#info_refs as /
Processing by Projects::GitHttpController#info_refs as /
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"fbucek", "project_id"=>"code-management.git"}
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"jobs", "project_id"=>"job-tasks.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 20ms (Views: 0.6ms | ActiveRecord: 4.1ms)
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 62ms (Views: 1.1ms | ActiveRecord: 3.1ms)
Started GET "/fbucek/code-management.git/info/refs?service=git-upload-pack" for 172.18.0.1 at 2019-05-07 16:44:27 +0000
Processing by Projects::GitHttpController#info_refs as /
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"fbucek", "project_id"=>"code-management.git"}
Started GET "/jobs/job-tasks.git/info/refs?service=git-upload-pack" for 172.18.0.1 at 2019-05-07 16:44:27 +0000
Processing by Projects::GitHttpController#info_refs as /
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"jobs", "project_id"=>"job-tasks.git"}
Completed 200 OK in 255ms (Views: 0.4ms | ActiveRecord: 20.5ms)
Completed 200 OK in 446ms (Views: 0.4ms | ActiveRecord: 39.6ms)
Started GET "/-/metrics" for 127.0.0.1 at 2019-05-07 16:44:29 +0000
Processing by MetricsController#index as HTML
Completed 200 OK in 32ms (Views: 0.7ms | ActiveRecord: 0.0ms)
Started GET "/-/metrics" for 127.0.0.1 at 2019-05-07 16:44:44 +0000
Processing by MetricsController#index as HTML
Completed 200 OK in 129ms (Views: 0.8ms | ActiveRecord: 0.0ms)
tail: '/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-04-29-14-06-10.log' has become inaccessible: Too many open files
tail: '/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-01-24-07-19-39.log' has become accessible


==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
I, [2019-05-07T12:33:04.600828 #638]  INFO -- : Refreshing Gem list
I, [2019-05-07T12:34:12.609267 #638]  INFO -- : listening on addr=127.0.0.1:8080 fd=21
I, [2019-05-07T12:34:12.610486 #638]  INFO -- : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I, [2019-05-07T12:34:12.613207 #638]  INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=22
I, [2019-05-07T12:34:12.734492 #638]  INFO -- : master process ready
I, [2019-05-07T12:34:12.836517 #855]  INFO -- : worker=0 ready
I, [2019-05-07T12:34:12.864967 #861]  INFO -- : worker=2 ready
I, [2019-05-07T12:34:12.885325 #858]  INFO -- : worker=1 ready
I, [2019-05-07T12:34:12.927121 #864]  INFO -- : worker=3 ready
W, [2019-05-07T12:37:45.732255 #864]  WARN -- : #Unicorn::HttpServer:0x00007f980beb7260: worker (pid: 864) exceeds memory limit (448656384.0 bytes > 437936556 bytes)
W, [2019-05-07T12:37:45.733867 #864]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 864) alive: 212 sec (trial 1)
I, [2019-05-07T12:37:46.107825 #638]  INFO -- : reaped #<Process::Status: pid 864 exit 0> worker=3
I, [2019-05-07T12:37:46.642380 #2085]  INFO -- : worker=3 ready
I, [2019-05-07T12:50:10.745083 #638]  INFO -- : reaped #<Process::Status: pid 858 exit 0> worker=1
I, [2019-05-07T12:50:10.747053 #638]  INFO -- : reaped #<Process::Status: pid 861 exit 0> worker=2
I, [2019-05-07T12:50:10.747986 #638]  INFO -- : reaped #<Process::Status: pid 2085 exit 0> worker=3
I, [2019-05-07T12:50:52.562351 #770]  INFO -- : Refreshing Gem list
I, [2019-05-07T12:51:29.639729 #770]  INFO -- : listening on addr=127.0.0.1:8080 fd=21
I, [2019-05-07T12:51:29.641049 #770]  INFO -- : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I, [2019-05-07T12:51:29.644441 #770]  INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=22
I, [2019-05-07T12:51:29.818334 #770]  INFO -- : master process ready
I, [2019-05-07T12:51:30.022666 #931]  INFO -- : worker=2 ready
I, [2019-05-07T12:51:30.048932 #934]  INFO -- : worker=3 ready
I, [2019-05-07T12:51:30.048932 #925]  INFO -- : worker=0 ready
I, [2019-05-07T12:51:30.077419 #928]  INFO -- : worker=1 ready
W, [2019-05-07T13:44:01.101729 #934]  WARN -- : #Unicorn::HttpServer:0x00007fcbd3c9f530: worker (pid: 934) exceeds memory limit (506458624.0 bytes > 502553016 bytes)
W, [2019-05-07T13:44:01.105212 #934]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 934) alive: 3127 sec (trial 1)
I, [2019-05-07T13:44:01.390238 #770]  INFO -- : reaped #<Process::Status: pid 934 exit 0> worker=3
I, [2019-05-07T13:44:01.692976 #8960]  INFO -- : worker=3 ready
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
caught error of type Gitlab::Git::CommandError in after callback inside Grape::Middleware::Formatter : 2:read info line: EOF
tail: '/var/log/gitlab/nginx/state' has become inaccessible: Too many open files
tail: '/var/log/gitlab/nginx/access.log' has become accessible

Details of package version

Provide the package version installation details 
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                       Version                    Architecture               Description
+++-==========================================-==========================-==========================-=========================================================================================
ii  gitlab-ce                                  11.10.4-ce.0               amd64                      GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                                                                            (no description available)

Environment details

  • Operating System: macOS
  • Installation Target, remove incorrect values:
    • Other: Docker Desktop version: 2.0.0.3
  • Installation Type, remove incorrect values:
    • Upgrade continuously from versions 10.10.0 / 10.9.0 / 10.8.0 ( always the same problem )
  • Is there any other software running on the machine: Running alone in Docker
  • Is this a single or multiple node installation? Single
  • Resources
    • CPU: 2.5 GHZ Inter Core i5 - 4 core ( Docker assigned 3 cores )
    • Memory total: 12 GB ( Docker 6GB )

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
external_url 'http://imac.local:9080'
postgresql['max_files_per_process'] = 200
nginx['http2_enabled'] = true
pages_external_url "http://pages.imac.dev/"
gitlab_pages['enable'] = true
gitlab_pages['redirect_http'] = false
gitlab_pages['use_http2'] = true
gitlab_pages['inplace_chroot'] = true