GDK setup fails with 502 Bad Gateway
Problem
Users get a You can access your GDK here message with a link, but when they click it, they see a 502 error because GDK setup actually failed.
Root cause
Running bash -x support/migrate-praefect shows this error:
Click to expand
+ set -e
++ gdk config get praefect.__praefect_build_bin_path
⚠️ WARNING: GDK setup in progress...
+ /projects/gitlab-development-kit/gitaly/_build/bin/praefect Run '`tail' -f '/projects/workspace-logs/poststart-stdout.log`' to watch the progress. -config gitaly/praefect.config.toml sql-migrate
Incorrect Usage: flag provided but not defined: -f
NAME:
praefect - a gitaly proxy
USAGE:
praefect [global options] [command [command options]]
VERSION:
Praefect, version 18.1.0-rc1-285-g6b4b2c195
COMMANDS:
serve launch the server daemon
configuration manage configuration
accept-dataloss accept potential data loss in a repository
check run startup checks
dataloss identify potential data loss in repositories
dial-nodes check connections
list-storages list physical storages for virtual storages
list-untracked-repositories list untracked repositories
track-repository start tracking a repository
track-repositories process bulk requests to track repositories
verify mark repository replicas as unverified
metadata show metadata about a repository
sql-ping check reachability of the database
sql-migrate apply outstanding SQL migrations
sql-migrate-down apply revert SQL migrations
sql-migrate-status show applied database migrations
remove-repository remove a repository
set-replication-factor set a replication factor for a repository
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--config FILE load configuration from FILE
--help, -h show help
--version, -v print the version
flag provided but not defined: -f
The gdk config get command should return only the praefect path, but it's also printing a warning message. Praefect fails because it receives -f (from the warning text "tail -f") as an invalid argument.
This warning was introduced by !5043 (merged).
Impact
Users can't use their workspaces because GDK never fully starts, even though it looks like setup finished successfully. The setup fails at the update_gdk step when running praefect migrations.
Proposal
Fix the check_workspace_setup_complete method that's printing warnings during command substitution. Either skip warnings for gdk config get commands or output warnings to stderr only.
Impacted categories
The following categories relate to this issue:
-
gdk-reliability - e.g. When a GDK action fails to complete. -
gdk-usability - e.g. Improvements or suggestions around how the GDK functions. -
gdk-performance - e.g. When a GDK action is slow or times out.
