gitlab-http-router does not come up in workspace
When spinning up a workspace, the gitlab-http-router doesn't come up.
Running gdk restart gitlab-http-router does always time out.
Looking at the process list, it kinda makes sense.
gitlab-+ 15023 0.0 0.0 2668 768 ? Ss 06:24 0:06 runsv gitlab-http-router
gitlab-+ 31837 0.0 0.0 2824 768 ? S 06:34 0:00 runsvdir -P /projects/gitlab-development-kit/services log: tp-router: fatal: unable to lock supervise/lock: temporary failure runsv gitlab-http-router: fatal: unable to lock supervise/lock: temporary failure runsv gitlab-http-router: fatal: unable to lock supervise/lock: temporary failure runsv gitlab-http-router: fatal: unable to lock supervise/lock: temporary failure runsv gitlab-http-router: fatal: unable to lock supervise/lock: temporary failure
Steps to replicate (optional)
- Spin up a new workspace (
mainbranch should suffice) - Visit the GDK and observe 502 Gateway errors
-
gdk statusand see that the router is down -
gdk start gitlab-http-routerand observe the timeout -
make gitlab-http-router-setup, because maybe something had gone wrong there -
ps aux | grep routerand observe the error shown above - Manually start the
gitlab-http-routerby executing what is written in the service config (afterexec-cd) to see that this is not necessarily a problem with the router itself. - Kill both the processes you see above (at least I did that)
- Then run
gdk start http-router - Observe that the router works now and GDK is accessible
🎉
Environment
General
-
Operating System:
Linux 6.8.0-1024-gke #28-Ubuntu SMP Tue Apr 15 21:31:25 UTC 2025 x86_64 x86_64 GNU/Linux -
Architecture:
x86_64 -
Ruby Version:
ruby 3.3.8 (2025-04-09 revision b200bad6cd) [x86_64-linux] -
GDK version:
632d9660 -
Package Manager:
mise-en-place 2025.6.7 linux-x64 (2025-06-23)
Variables
PATH=$HOME/.local/share/mise/installs/node/20.12.2/bin:$HOME/.local/share/mise/installs/yarn/1.22.19/bin:$HOME/.local/share/mise/installs/redis/7.0.14/bin:$HOME/.local/share/mise/installs/minio/2022-07-15T03-44-22Z/bin:$HOME/.local/share/mise/installs/postgres/16.8/bin:$HOME/.local/share/mise/installs/postgres/14.9/bin:$HOME/.cargo/bin:$HOME/.local/share/mise/installs/ruby/3.3.8/bin:$HOME/.local/share/mise/installs/ruby/3.2.4/bin:$HOME/.local/share/mise/installs/shellcheck/0.10.0:$HOME/.local/share/mise/installs/markdownlint-cli2/0.18.1/bin:$HOME/.local/share/mise/installs/vale/3.11.2:$HOME/.local/share/mise/installs/go/1.24.4/bin:/projects/.gl-tools/vscode-reh-web/bin/remote-cli:$HOME/.local/bin:$HOME/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=en_US.UTF-8
LANGUAGE=
LC_ALL=
LDFLAGS=
CPPFLAGS=
PKG_CONFIG_PATH=
LIBPCREDIR=
RUBY_CONFIGURE_OPTS=
GDK Configuration
---
asdf:
opt_out: true
gitaly:
skip_setup: true
gitlab:
rails:
hostname: 3000-workspace-73241-27176846-howzjw.workspaces.gitlab.dev
https:
enabled: true
port: 443
gitlab_shell:
skip_setup: true
listen_address: 10.20.3.35
mise:
enabled: true
telemetry:
enabled: true
environment: remote-development
username: "[redacted]"
vite:
enabled: true
hot_module_reloading: false
webpack:
enabled: false
GDK Doctor
Result
..............E...WW......W......W...
⚠️ WARNING: Your GDK may need attention.
[Correctable] Git Maintenance Recommendation
================================================================================
We recommend enabling git-maintenance to avoid slowdowns of local git operations like fetch, pull, and checkout.
To enable it, run `git maintenance start` in each repository:
git -C /projects/gitlab-development-kit maintenance start
git -C /projects/gitlab-development-kit/gitlab maintenance start
Missing Binaries
================================================================================
The following binaries are missing from their expected paths:
gitaly
Please ensure you download them by running:
make gitaly-setup
Port required by GDK is already in use
================================================================================
No such file or directory - lsof
$HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/3.3.0/open3.rb:534:in `spawn'
$HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/3.3.0/open3.rb:534:in `popen_run'
$HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/3.3.0/open3.rb:235:in `popen3'
$HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/3.3.0/open3.rb:658:in `capture3'
/projects/gitlab-development-kit/lib/gdk/shellout.rb:160:in `capture'
/projects/gitlab-development-kit/lib/gdk/shellout.rb:108:in `run'
/projects/gitlab-development-kit/lib/gdk/diagnostic/port_in_use.rb:52:in `run'
/projects/gitlab-development-kit/lib/gdk/diagnostic/port_in_use.rb:34:in `port_in_use?'
/projects/gitlab-development-kit/lib/gdk/diagnostic/port_in_use.rb:9:in `success?'
/projects/gitlab-development-kit/lib/gdk/command/doctor.rb:61:in `perform_diagnosis_for'
/projects/gitlab-development-kit/lib/gdk/command/doctor.rb:54:in `block (2 levels) in jobs'
Stale Data
================================================================================
You might encounter a PG::CheckViolation error during database migrations, likely due to stale data in the ci database that belongs in the main database, or vice versa. To address this, you can run:
gdk truncate-legacy-tables
GDK Status
================================================================================
The following services are not running but should be:
🔻 down → Log file not found for down
Ruby Environment
Gem Environment
RubyGems Environment:
- RUBYGEMS VERSION: 3.6.9
- RUBY VERSION: 3.3.8 (2025-04-09 patchlevel 144) [x86_64-linux]
- INSTALLATION DIRECTORY: $HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/gems/3.3.0
- USER INSTALLATION DIRECTORY: $HOME/.local/share/gem/ruby/3.3.0
- CREDENTIALS FILE: $HOME/.local/share/gem/credentials
- RUBY EXECUTABLE: $HOME/.local/share/mise/installs/ruby/3.3.8/bin/ruby
- GIT EXECUTABLE: /usr/bin/git
- EXECUTABLE DIRECTORY: $HOME/.local/share/mise/installs/ruby/3.3.8/bin
- SPEC CACHE DIRECTORY: $HOME/.cache/gem/specs
- SYSTEM CONFIGURATION DIRECTORY: $HOME/.local/share/mise/installs/ruby/3.3.8/etc
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- $HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/gems/3.3.0
- $HOME/.local/share/gem/ruby/3.3.0
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :backtrace => true
- :bulk_threshold => 1000
- REMOTE SOURCES:
- https://rubygems.org/
- SHELL PATH:
- $HOME/.local/share/mise/installs/node/20.12.2/bin
- $HOME/.local/share/mise/installs/yarn/1.22.19/bin
- $HOME/.local/share/mise/installs/redis/7.0.14/bin
- $HOME/.local/share/mise/installs/minio/2022-07-15T03-44-22Z/bin
- $HOME/.local/share/mise/installs/postgres/16.8/bin
- $HOME/.local/share/mise/installs/postgres/14.9/bin
- $HOME/.cargo/bin
- $HOME/.local/share/mise/installs/ruby/3.3.8/bin
- $HOME/.local/share/mise/installs/ruby/3.2.4/bin
- $HOME/.local/share/mise/installs/shellcheck/0.10.0
- $HOME/.local/share/mise/installs/markdownlint-cli2/0.18.1/bin
- $HOME/.local/share/mise/installs/vale/3.11.2
- $HOME/.local/share/mise/installs/go/1.24.4/bin
- /projects/.gl-tools/vscode-reh-web/bin/remote-cli
- $HOME/.local/bin
- $HOME/.local/bin
- /usr/local/sbin
- /usr/local/bin
- /usr/sbin
- /usr/bin
- /sbin
- /bin
Bundle Environment
Environment
Bundler 2.6.9
Platforms ruby, x86_64-linux
Ruby 3.3.8p144 (2025-04-09 revision b200bad6cd40d08e9f33b93e1a85c270b337867c) [x86_64-linux]
Full Path $HOME/.local/share/mise/installs/ruby/3.3.8/bin/ruby
Config Dir $HOME/.local/share/mise/installs/ruby/3.3.8/etc
RubyGems 3.6.9
Gem Home $HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/gems/3.3.0
Gem Path $HOME/.local/share/gem/ruby/3.3.0:$HOME/.local/share/mise/installs/ruby/3.3.8/lib/ruby/gems/3.3.0
User Home $HOME
User Path $HOME/.local/share/gem/ruby/3.3.0
Bin Dir $HOME/.local/share/mise/installs/ruby/3.3.8/bin
Tools
Git 2.34.1
RVM not installed
rbenv not installed
chruby not installed
Bundler Build Metadata
Built At 1980-01-02
Git SHA 8a2a14d63da
Released Version true
Gemfile
Gemfile
# frozen_string_literal: true
source 'https://rubygems.org'
gemspec path: 'gem/'
group :development do
gem 'lefthook', '~> 1.10.10', require: false
gem 'rubocop', require: false
gem "rubocop-rake", "~> 0.6.0", require: false
gem 'yard', '~> 0.9.37', require: false
gem 'pry-byebug' # See doc/howto/pry.md
end
group :test do
gem 'gitlab-styles', '~> 13.0.2', require: false
gem 'irb', '~> 1.15.1', require: false
gem 'rspec', '~> 3.13.0', require: false
gem 'rspec_junit_formatter', '~> 0.6.0', require: false
gem 'simplecov-cobertura', '~> 2.1.0', require: false
gem 'webmock', '~> 3.25', require: false
end
group :development, :test, :danger do
gem 'gitlab-dangerfiles', '~> 4.8.1', require: false
gem 'resolv', '~> 0.6.0', require: false
gem 'ruby-lsp', "~> 0.23.0", require: false
gem 'ruby-lsp-rspec', "~> 0.1.10", require: false
end
Gemfile.lock
PATH
remote: gem
specs:
gitlab-development-kit (0.2.19)
gitlab-sdk (~> 0.3.1)
rake (~> 13.1)
sentry-ruby (~> 5.23)
tty-markdown (~> 0.7.2)
tty-spinner (~> 0.9.3)
zeitwerk (~> 2.6.15)
GEM
remote: https://rubygems.org/
specs:
activesupport (7.1.3.4)
base64
bigdecimal
concurrent-ruby (~> 1.0, >= 1.0.2)
connection_pool (>= 2.2.5)
drb
i18n (>= 1.6, < 2)
minitest (>= 5.1)
mutex_m
tzinfo (~> 2.0)
addressable (2.8.7)
public_suffix (>= 2.0.2, < 7.0)
ast (2.4.2)
base64 (0.2.0)
bigdecimal (3.1.8)
byebug (11.1.3)
claide (1.1.0)
claide-plugins (0.9.2)
cork
nap
open4 (~> 1.3)
coderay (1.1.3)
colored2 (3.1.2)
concurrent-ruby (1.3.3)
connection_pool (2.4.1)
cork (0.3.0)
colored2 (~> 3.1)
crack (1.0.0)
bigdecimal
rexml
csv (3.3.0)
danger (9.4.3)
claide (~> 1.0)
claide-plugins (>= 0.9.2)
colored2 (~> 3.1)
cork (~> 0.1)
faraday (>= 0.9.0, < 3.0)
faraday-http-cache (~> 2.0)
git (~> 1.13)
kramdown (~> 2.3)
kramdown-parser-gfm (~> 1.0)
no_proxy_fix
octokit (>= 4.0)
terminal-table (>= 1, < 4)
danger-gitlab (8.0.0)
danger
gitlab (~> 4.2, >= 4.2.0)
diff-lcs (1.5.1)
docile (1.4.0)
drb (2.2.1)
faraday (2.9.2)
faraday-net_http (>= 2.0, < 3.2)
faraday-http-cache (2.5.1)
faraday (>= 0.8)
faraday-net_http (3.1.0)
net-http
git (1.19.1)
addressable (~> 2.8)
rchardet (~> 1.8)
gitlab (4.20.1)
httparty (~> 0.20)
terminal-table (>= 1.5.1)
gitlab-dangerfiles (4.8.1)
danger (>= 9.3.0)
danger-gitlab (>= 8.0.0)
rake (~> 13.0)
gitlab-sdk (0.3.1)
activesupport (>= 5.2.0)
rake (~> 13.0)
snowplow-tracker (~> 0.8.0)
gitlab-styles (13.0.2)
rubocop (~> 1.68.0)
rubocop-capybara (~> 2.21.0)
rubocop-factory_bot (~> 2.26.1)
rubocop-graphql (~> 1.5.4)
rubocop-performance (~> 1.21.1)
rubocop-rails (~> 2.26.0)
rubocop-rspec (~> 3.0.4)
rubocop-rspec_rails (~> 2.30.0)
hashdiff (1.1.1)
httparty (0.22.0)
csv
mini_mime (>= 1.0.0)
multi_xml (>= 0.5.2)
i18n (1.14.5)
concurrent-ruby (~> 1.0)
io-console (0.7.2)
irb (1.15.1)
pp (>= 0.6.0)
rdoc (>= 4.0.0)
reline (>= 0.4.2)
json (2.7.2)
kramdown (2.4.0)
rexml
kramdown-parser-gfm (1.1.0)
kramdown (~> 2.0)
language_server-protocol (3.17.0.3)
lefthook (1.10.10)
logger (1.6.6)
method_source (1.0.0)
mini_mime (1.1.5)
minitest (5.24.1)
multi_xml (0.7.1)
bigdecimal (~> 3.1)
mutex_m (0.2.0)
nap (1.1.0)
net-http (0.4.1)
uri
no_proxy_fix (0.1.2)
octokit (6.1.1)
faraday (>= 1, < 3)
sawyer (~> 0.9)
open4 (1.3.4)
parallel (1.25.1)
parser (3.3.3.0)
ast (~> 2.4.1)
racc
pastel (0.8.0)
tty-color (~> 0.5)
pp (0.6.2)
prettyprint
prettyprint (0.2.0)
prism (1.3.0)
pry (0.14.2)
coderay (~> 1.1)
method_source (~> 1.0)
pry-byebug (3.10.1)
byebug (~> 11.0)
pry (>= 0.13, < 0.15)
psych (5.1.2)
stringio
public_suffix (5.1.1)
racc (1.8.0)
rack (3.1.8)
rainbow (3.1.1)
rake (13.2.1)
rbs (3.8.1)
logger
rchardet (1.8.0)
rdoc (6.6.3.1)
psych (>= 4.0.0)
regexp_parser (2.9.2)
reline (0.5.9)
io-console (~> 0.5)
resolv (0.6.0)
rexml (3.3.1)
strscan
rouge (4.4.0)
rspec (3.13.0)
rspec-core (~> 3.13.0)
rspec-expectations (~> 3.13.0)
rspec-mocks (~> 3.13.0)
rspec-core (3.13.0)
rspec-support (~> 3.13.0)
rspec-expectations (3.13.0)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.13.0)
rspec-mocks (3.13.0)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.13.0)
rspec-support (3.13.0)
rspec_junit_formatter (0.6.0)
rspec-core (>= 2, < 4, != 2.12.0)
rubocop (1.68.0)
json (~> 2.3)
language_server-protocol (>= 3.17.0)
parallel (~> 1.10)
parser (>= 3.3.0.2)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 2.4, < 3.0)
rubocop-ast (>= 1.32.2, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 3.0)
rubocop-ast (1.32.3)
parser (>= 3.3.1.0)
rubocop-capybara (2.21.0)
rubocop (~> 1.41)
rubocop-factory_bot (2.26.1)
rubocop (~> 1.61)
rubocop-graphql (1.5.4)
rubocop (>= 1.50, < 2)
rubocop-performance (1.21.1)
rubocop (>= 1.48.1, < 2.0)
rubocop-ast (>= 1.31.1, < 2.0)
rubocop-rails (2.26.2)
activesupport (>= 4.2.0)
rack (>= 1.1)
rubocop (>= 1.52.0, < 2.0)
rubocop-ast (>= 1.31.1, < 2.0)
rubocop-rake (0.6.0)
rubocop (~> 1.0)
rubocop-rspec (3.0.5)
rubocop (~> 1.61)
rubocop-rspec_rails (2.30.0)
rubocop (~> 1.61)
rubocop-rspec (~> 3, >= 3.0.1)
ruby-lsp (0.23.11)
language_server-protocol (~> 3.17.0)
prism (>= 1.2, < 2.0)
rbs (>= 3, < 4)
sorbet-runtime (>= 0.5.10782)
ruby-lsp-rspec (0.1.22)
ruby-lsp (~> 0.23.0)
ruby-progressbar (1.13.0)
sawyer (0.9.2)
addressable (>= 2.3.5)
faraday (>= 0.17.3, < 3)
sentry-ruby (5.23.0)
bigdecimal
concurrent-ruby (~> 1.0, >= 1.0.2)
simplecov (0.21.2)
docile (~> 1.1)
simplecov-html (~> 0.11)
simplecov_json_formatter (~> 0.1)
simplecov-cobertura (2.1.0)
rexml
simplecov (~> 0.19)
simplecov-html (0.12.3)
simplecov_json_formatter (0.1.4)
snowplow-tracker (0.8.0)
sorbet-runtime (0.5.11911)
stringio (3.1.1)
strings (0.2.1)
strings-ansi (~> 0.2)
unicode-display_width (>= 1.5, < 3.0)
unicode_utils (~> 1.4)
strings-ansi (0.2.0)
strscan (3.1.0)
terminal-table (3.0.2)
unicode-display_width (>= 1.1.1, < 3)
tty-color (0.6.0)
tty-cursor (0.7.1)
tty-markdown (0.7.2)
kramdown (>= 1.16.2, < 3.0)
pastel (~> 0.8)
rouge (>= 3.14, < 5.0)
strings (~> 0.2.0)
tty-color (~> 0.5)
tty-screen (~> 0.8)
tty-screen (0.8.2)
tty-spinner (0.9.3)
tty-cursor (~> 0.7)
tzinfo (2.0.6)
concurrent-ruby (~> 1.0)
unicode-display_width (2.5.0)
unicode_utils (1.4.0)
uri (0.13.0)
webmock (3.25.0)
addressable (>= 2.8.0)
crack (>= 0.3.2)
hashdiff (>= 0.4.0, < 2.0.0)
yard (0.9.37)
zeitwerk (2.6.15)
PLATFORMS
ruby
DEPENDENCIES
gitlab-dangerfiles (~> 4.8.1)
gitlab-development-kit!
gitlab-styles (~> 13.0.2)
irb (~> 1.15.1)
lefthook (~> 1.10.10)
pry-byebug
resolv (~> 0.6.0)
rspec (~> 3.13.0)
rspec_junit_formatter (~> 0.6.0)
rubocop
rubocop-rake (~> 0.6.0)
ruby-lsp (~> 0.23.0)
ruby-lsp-rspec (~> 0.1.10)
simplecov-cobertura (~> 2.1.0)
webmock (~> 3.25)
yard (~> 0.9.37)
BUNDLED WITH
2.6.9
Gemspecs
gitlab-development-kit.gemspec
# frozen_string_literal: true
$LOAD_PATH.unshift(File.expand_path('lib', __dir__))
require 'gitlab_development_kit'
Gem::Specification.new do |spec|
spec.name = 'gitlab-development-kit'
spec.version = GDK::GEM_VERSION
spec.authors = ['Jacob Vosmaer', 'GitLab']
spec.email = ['gitlab_rubygems@gitlab.com']
spec.summary = 'CLI for GitLab Development Kit'
spec.description = 'CLI for GitLab Development Kit.'
spec.homepage = 'https://gitlab.com/gitlab-org/gitlab-development-kit'
spec.license = 'MIT'
spec.files = ['lib/gitlab_development_kit.rb']
spec.executables = ['gdk']
spec.required_ruby_version = '>= 3.2.0'
spec.metadata['rubygems_mfa_required'] = 'true'
spec.add_dependency 'gitlab-sdk', '~> 0.3.1'
spec.add_dependency 'rake', '~> 13.1'
spec.add_dependency 'sentry-ruby', '~> 5.23'
spec.add_dependency 'tty-markdown', '~> 0.7.2'
spec.add_dependency 'tty-spinner', '~> 0.9.3'
spec.add_dependency 'zeitwerk', '~> 2.6.15'
end
Telemetry
Network information
Unknown (No such file or directory - lsof)
Logs
Git Repositories
Git Status and HEAD for Gdk
Git Status:
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
index.html
nothing added to commit but untracked files present (use "git add" to track)
Git HEAD:
commit 632d966069c1515a21310f58c9566ff4471e7e19
Git Status and HEAD for Gitaly
Git Status:
HEAD detached at 3de474413
nothing to commit, working tree clean
Git HEAD:
commit 3de4744135aca47419474d244258a5e285dad834
Git Status and HEAD for Gitlab
Git Status:
On branch master
Your branch is up to date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: db/docs/abuse_report_uploads.yml
modified: db/docs/achievement_uploads.yml
modified: db/docs/ai_vectorizable_file_uploads.yml
modified: db/docs/alert_management_alert_metric_image_uploads.yml
modified: db/docs/appearance_uploads.yml
modified: db/docs/bulk_import_export_upload_uploads.yml
modified: db/docs/dependency_list_export_part_uploads.yml
modified: db/docs/dependency_list_export_uploads.yml
modified: db/docs/design_management_action_uploads.yml
modified: db/docs/import_export_upload_uploads.yml
modified: db/docs/issuable_metric_image_uploads.yml
modified: db/docs/namespace_uploads.yml
modified: db/docs/note_uploads.yml
modified: db/docs/organization_detail_uploads.yml
modified: db/docs/project_import_export_relation_export_upload_uploads.yml
modified: db/docs/project_topic_uploads.yml
modified: db/docs/project_uploads.yml
modified: db/docs/snippet_uploads.yml
modified: db/docs/uploads_9ba88c4165.yml
modified: db/docs/user_permission_export_upload_uploads.yml
modified: db/docs/user_uploads.yml
modified: db/docs/vulnerability_archive_export_uploads.yml
modified: db/docs/vulnerability_export_part_uploads.yml
modified: db/docs/vulnerability_export_uploads.yml
modified: db/docs/vulnerability_remediation_uploads.yml
modified: db/structure.sql
no changes added to commit (use "git add" and/or "git commit -a")
Git HEAD:
commit 566e663aa8daeacd21704a13e14f953b72207ed2
Created at: 24/06/2025 07:42:47 UTC
Proposal
Might be a long shot, but maybe we are copying runsv-related state over from the bootstrapped image.
I did not verify that, but if that's the case we should probably do a cleanup. This looks a lot like the various remarks in Slack (search for gdk kill) indicating that GDK can get stuck in a state where a service was not successfully shutdown. If such a situation can arise during building the workspace image, and this is related to runsv , the workspace would basically start in such a "stuck" situation".