constraint errors after update from 15.11 to 16.10 (PG::CheckViolation: ERROR: new row for relation "users" violates check constraint "check_0dd5948e38")
I am having issues after updating from 15.11 to 16.10 a Gitlab CE omnibus installation - it throws 500 errors for almost everybody, it is having issues with a constraint on the users table (I removed the username, email, personal information...):
ActiveRecord::StatementInvalid (PG::CheckViolation: ERROR: new row for relation "users" violates check constraint "check_0dd5948e38"
DETAIL: Failing row contains (57, <USER_EMAIL>, <USER_PASSWORD>, null, null, null, 149, 2023-06-23 12:21:38.033162, 2023-06-22 12:48:36.622267, 79.121.25.166, 79.121.25.166, 2021-03-04 07:05:25.635024, 2023-06-26 08:33:34.315288, <USER_NAME>, f, 100000, 0, null, <USERNAME>, t, f, active, 1, null, 3, null, null, null, 2021-03-04 07:05:25.191663, null, null, f, <USER_EMAIL>, f, f, null, null, null, f, null, , 0, 2, null, 0, f, null, null, f, 2p88n4zc47cgoo0zsruwrxh87, f, 48, 2023-06-26, f, en, 1, null, null, t8nzwep_izqJcK4ZwXdn, f, null, f, null, null, null, null, null, null, null, , null, null, null, null, 0, null, null, null, f).
):
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
app/services/users/activity_service.rb:40:in `record_activity'
app/services/users/activity_service.rb:22:in `block in execute'
lib/gitlab/database/load_balancing/session.rb:55:in `ignore_writes'
lib/gitlab/database/load_balancing/session.rb:23:in `without_sticky_writes'
app/services/users/activity_service.rb:22:in `execute'
app/controllers/concerns/record_user_last_activity.rb:24:in `set_user_last_activity'
app/controllers/application_controller.rb:500:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:491:in `set_session_storage'
lib/gitlab/i18n.rb:107:in `with_locale'
lib/gitlab/i18n.rb:113:in `with_user_locale'
app/controllers/application_controller.rb:482:in `set_locale'
app/controllers/application_controller.rb:475:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Users cannot log in at the moment, but logged in users are mostly okay - some features are problematic (cannot disable new navigation, etc..)
Gitlab check can be seen below, migrations ran fine without errors, upgrade experience was smooth, no visible problems.
# sudo gitlab-rake gitlab:check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
<PROJECTS_LIST>...
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 4
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished