web_1 failing to start with active_record/dynamic_matchers.rb:22:in `method_missing' Early termination of worker
Problem
The web server is not starting and the following is repeated in the logs:
web_1 | [2644] ! Unable to start worker
web_1 | [2644] /data/cache/bundle-2.7.4/ruby/2.7.0/gems/activerecord-6.1.4.4/lib/active_record/dynamic_matchers.rb:22:in `method_missing'
web_1 | [2644] Early termination of worker
More detail
From this internal slack thread https://gitlab.slack.com/archives/CDPA9TK1B/p1644504295523169
Further debugging editing https://github.com/puma/puma/blob/v5.5.2/lib/puma/cluster/worker.rb#L61 led to https://gitlab.com/gitlab-org/gitlab/-/snippets/2252124 indicating:
heinrich: I see the problem now. The User model is being loaded in 7_prometheus_metrics which is before 8_devise. The user model needs devise for the devise call on that line the stacktrace is pointing to ...
So renaming config/7_prometheus_metrics.rb to config/8_prometheus_metrics.rb should also work I think ...
Part of the problem is that if puma is not in clustered mode, it executes the on_master_start block inline. Maybe we should defer this after boot to make it similar?
Workaround
Open a rails console with gck shell
then bin/rails c
and you can workaround the problem by disabling prometheus metrics:
ApplicationSetting.current.update!(prometheus_metrics_enabled: false)