Transaction timeout during migration in validate_foreign_key 15.9.x
Summary
During an update from 15.8.4 to 15.9.3 and later 15.9.0 (official docker images gitlab/gitlab-ce [15.9.3-ce.0, 15.9.0-ce.0]) we got an error regarding a transaction beeing failed. This leads to an error during update and we had to revert back to 15.8.4-ce.0 image. This works fine now.
The error states that in /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:363:in the validate_foreign_key' method fails called in up method of /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:9:in up'.
After some investigation we found out that 20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb calls validate_foreign_key with colums as arguments which we think is not supported at the moment (This should be addressed in .16 release).
20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:
....
COLUMNS = [:partition_id, :build_id]
def up
validate_foreign_key(TABLE_NAME, COLUMNS, name: FK_NAME)
end
...https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/database/migration_helpers.rb uses only column as argument
def validate_foreign_key(source, column, name: nil)
fk_name = name || concurrent_foreign_key_name(source, column)Steps to reproduce
In our setup we use the official docker image and mount some host paths. The setup is the same for every version of gitlab so basically we exchange the docker image version numbers like: docker-compose.yml:
gitlab:
image: "gitlab/gitlab-ce:15.8.4-ce.0" # exchange the versions like '15.9.0-ce.0 or 15.9.3-ce.0' both fail, #15.8.4-ce.0' works.
restart: always
hostname: "some.hostname.tld"
volumes:
- "{{ some host path }}/config:/etc/gitlab"
- "{{ some host path }}/logs:/var/log/gitlab"
- "{{ some host path }}/data:/var/opt/gitlab"
depends_on:
other containers that do credentials for gitlab
shm_size: '256m'
networks:
- front
- back
What is the current bug behavior?
If we spin up the with docker-compose up and the docker-compose.yml file with the gitlab service configured like above. When we change the version to anything >= 15.9.0 we get an error stating that:
PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction blockThis is because:
Caused by:
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR: insert or update on table "ci_builds_runner_session" violates foreign key constraint "fk_rails_70707857d3_p"
DETAIL: Key (partition_id, build_id)=(100, 1918079) is not present in table "ci_builds".
... migration_helpers.rb:364:in `block in validate_foreign_key'
...
... migration_helpers.rb:363:in `validate_foreign_key'
... 20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:9:in `up'which presumably calls validate_foreign_key with a multi column fk . Which validate_foreign_key maybe not implements.
What is the expected correct behavior?
When the docker containers with the new version spin up they should do migrations and checks against the database with no errors.
Relevant logs and/or screenshots
gitlab_1 | [2023-03-20T21:03:55+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
gitlab_1 | [2023-03-20T21:03:55+00:00] FATAL: ---------------------------------------------------------------------------------------
gitlab_1 | [2023-03-20T21:03:55+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
gitlab_1 | [2023-03-20T21:03:55+00:00] FATAL: ---------------------------------------------------------------------------------------
gitlab_1 | [2023-03-20T21:03:55+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab_1 | ---- Begin output of "bash" ----
gitlab_1 | STDOUT: rake aborted!
gitlab_1 | StandardError: An error has occurred, this and all later migrations canceled:
gitlab_1 |
gitlab_1 | PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:33:in `ensure in disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:33:in `disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:363:in `validate_foreign_key'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:9:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:347:in `block in transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:346:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 |
gitlab_1 | Caused by:
gitlab_1 | ActiveRecord::StatementInvalid: PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:33:in `ensure in disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:33:in `disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:363:in `validate_foreign_key'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:9:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:347:in `block in transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:346:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 |
gitlab_1 | Caused by:
gitlab_1 | PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:33:in `ensure in disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:33:in `disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:363:in `validate_foreign_key'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:9:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:347:in `block in transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:346:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 |
gitlab_1 | Caused by:
gitlab_1 | ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR: insert or update on table "ci_builds_runner_session" violates foreign key constraint "fk_rails_70707857d3_p"
gitlab_1 | DETAIL: Key (partition_id, build_id)=(100, 1918079) is not present in table "ci_builds".
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:364:in `block in validate_foreign_key'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:363:in `validate_foreign_key'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:9:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:347:in `block in transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:346:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 |
gitlab_1 | Caused by:
gitlab_1 | PG::ForeignKeyViolation: ERROR: insert or update on table "ci_builds_runner_session" violates foreign key constraint "fk_rails_70707857d3_p"
gitlab_1 | DETAIL: Key (partition_id, build_id)=(100, 1918079) is not present in table "ci_builds".
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:364:in `block in validate_foreign_key'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:363:in `validate_foreign_key'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb:9:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:347:in `block in transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:346:in `transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 | Tasks: TOP => db:migrate
gitlab_1 | (See full trace by running task with --trace)
gitlab_1 | main: == 20230119095024 ValidateFkOnCiBuildsRunnerSessionPartitionIdAndBuildId: migrating
gitlab_1 | main: -- execute("SET statement_timeout TO 0")
gitlab_1 | main: -> 0.0003s
gitlab_1 | main: -- execute("ALTER TABLE ci_builds_runner_session VALIDATE CONSTRAINT fk_rails_70707857d3_p;")
gitlab_1 | main: -- execute("RESET statement_timeout")
gitlab_1 | STDERR:
gitlab_1 | ---- End output of "bash" ----
gitlab_1 | Ran "bash" returned 1
For both 15.9.0 and 15.9.3 the output is the same, besides the time related things.
Results of GitLab environment info
obviously of the last running 15.8.4 version:
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.7.7p221 Gem Version: 3.1.6 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.8 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.8.4 Revision: 54f2a13176e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: https://git... HTTP Clone URL: https://git.../some-group/some-project.git SSH Clone URL: git@git...:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: openid_connect GitLab Shell Version: 14.15.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.15.0 ? ... OK (14.15.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 4/16 ... yes 28/57 ... yes 34/62 ... yes 44/112 ... yes 47/118 ... yes 47/119 ... yes 45/123 ... yes 46/125 ... yes 45/127 ... yes 44/132 ... yes 4/151 ... yes 32/160 ... yes 5/173 ... yes 62/193 ... yes 62/195 ... yes 62/205 ... yes 63/207 ... yes 7/217 ... yes 45/239 ... yes 4/248 ... yes 32/271 ... yes 244/272 ... yes 45/278 ... yes 62/301 ... yes 24/303 ... yes 117/331 ... yes 44/350 ... yes 75/353 ... yes 117/363 ... yes 32/365 ... yes 117/370 ... yes 4/376 ... yes 39/382 ... yes 39/383 ... yes 80/389 ... yes 39/390 ... yes 86/391 ... yes 86/392 ... yes 118/395 ... yes 117/396 ... yes 65/398 ... yes 118/399 ... yes 86/401 ... yes 39/402 ... yes 86/403 ... yes 28/411 ... yes 62/416 ... yes 62/422 ... yes 24/426 ... yes 117/431 ... yes 62/435 ... yes 24/436 ... yes 24/437 ... yes 62/440 ... yes 88/443 ... yes 24/444 ... yes 24/450 ... yes 89/456 ... yes 24/457 ... yes 62/458 ... yes 24/459 ... yes 62/460 ... yes 62/463 ... yes 106/488 ... yes 28/490 ... yes 244/495 ... yes 62/496 ... yes 244/497 ... yes 62/503 ... yes 62/505 ... yes 62/507 ... yes 4/510 ... yes 24/512 ... yes 45/524 ... yes 24/527 ... yes 95/532 ... yes 118/536 ... yes 95/540 ... yes 95/541 ... yes 19/542 ... yes 83/548 ... yes 118/552 ... yes 4/556 ... yes 62/561 ... yes 44/562 ... yes 116/565 ... yes 113/566 ... yes 104/574 ... yes 65/576 ... yes 62/578 ... yes 4/579 ... yes 106/585 ... yes 106/586 ... yes 106/587 ... yes 106/588 ... yes 106/589 ... yes 106/590 ... yes 117/596 ... yes 117/624 ... yes 119/626 ... yes 117/627 ... yes 114/632 ... yes 106/633 ... yes 46/634 ... yes 46/635 ... yes 275/637 ... yes 46/646 ... yes 113/647 ... yes 46/649 ... yes 117/651 ... yes 62/658 ... yes 127/666 ... yes 117/667 ... yes 103/668 ... yes 24/669 ... yes 315/670 ... yes 117/671 ... yes 53/677 ... yes 113/680 ... yes 113/684 ... yes 114/685 ... yes 113/686 ... yes 119/688 ... yes 106/690 ... yes 95/693 ... yes 106/697 ... yes 113/706 ... yes 184/709 ... yes 112/710 ... yes 62/711 ... yes 117/714 ... yes 134/717 ... yes 249/721 ... yes 59/722 ... yes 117/723 ... yes 136/728 ... yes 136/729 ... yes 89/731 ... yes 137/732 ... yes 28/733 ... yes 62/738 ... yes 118/741 ... yes 117/742 ... yes 136/743 ... yes 62/744 ... yes 114/745 ... yes 62/746 ... yes 117/747 ... yes 118/748 ... yes 113/749 ... yes 113/750 ... yes 113/751 ... yes 114/752 ... yes 145/753 ... yes 113/755 ... yes 136/756 ... yes 4/757 ... yes 145/760 ... yes 117/763 ... yes 28/769 ... yes 115/774 ... yes 106/775 ... yes 106/777 ... yes 138/785 ... yes 62/789 ... yes 117/793 ... yes 145/794 ... yes 117/796 ... yes 139/797 ... yes 136/798 ... yes 62/799 ... yes 118/803 ... yes 117/804 ... yes 114/806 ... yes 32/807 ... yes 106/808 ... yes 106/809 ... yes 106/811 ... yes 305/813 ... yes 106/814 ... yes 134/815 ... yes 62/816 ... yes 136/818 ... yes 114/820 ... yes 225/824 ... yes 4/826 ... yes 106/827 ... yes 106/828 ... yes 106/829 ... yes 265/830 ... yes 151/834 ... yes 151/836 ... yes 106/837 ... yes 106/839 ... yes 117/840 ... yes 62/841 ... yes 106/842 ... yes 106/843 ... yes 4/844 ... yes 105/846 ... yes 106/850 ... yes 54/855 ... yes 117/858 ... yes 62/859 ... yes 24/860 ... yes 4/862 ... yes 118/863 ... yes 62/864 ... yes 155/865 ... yes 106/868 ... yes 117/870 ... yes 136/871 ... yes 106/872 ... yes 62/874 ... yes 66/876 ... yes 106/879 ... yes 136/883 ... yes 106/885 ... yes 163/888 ... yes 4/889 ... yes 24/890 ... yes 158/892 ... yes 281/893 ... yes 62/895 ... yes 279/896 ... yes 28/897 ... yes 24/899 ... yes 118/901 ... yes 136/903 ... yes 106/904 ... yes 162/908 ... yes 106/909 ... yes 158/910 ... yes 136/911 ... yes 106/912 ... yes 161/914 ... yes 28/920 ... yes 277/924 ... yes 186/925 ... yes 4/926 ... yes 106/928 ... yes 15/929 ... yes 277/930 ... yes 143/931 ... yes 133/932 ... yes 114/934 ... yes 106/935 ... yes 117/938 ... yes 117/940 ... yes 96/942 ... yes 161/948 ... yes 62/951 ... yes 62/952 ... yes 170/953 ... yes 28/954 ... yes 161/955 ... yes 134/957 ... yes 170/959 ... yes 113/960 ... yes 176/961 ... yes 199/962 ... yes 353/963 ... yes 184/964 ... yes 161/965 ... yes 260/966 ... yes 106/967 ... yes 187/977 ... yes 4/978 ... yes 176/981 ... yes 170/982 ... yes 15/985 ... yes 15/986 ... yes 176/987 ... yes 177/989 ... yes 98/992 ... yes 62/993 ... yes 106/994 ... yes 62/995 ... yes 176/996 ... yes 106/997 ... yes 177/998 ... yes 106/999 ... yes 113/1000 ... yes 13/1002 ... yes 155/1005 ... yes 166/1006 ... yes 289/1007 ... yes 34/1008 ... yes 106/1009 ... yes 139/1010 ... yes 289/1011 ... yes 155/1012 ... yes 99/1014 ... yes 177/1017 ... yes 114/1018 ... yes 328/1019 ... yes 259/1022 ... yes 114/1025 ... yes 199/1029 ... yes 62/1030 ... yes 106/1032 ... yes 206/1033 ... yes 106/1036 ... yes 139/1039 ... yes 177/1041 ... yes 307/1043 ... yes 307/1044 ... yes 93/1045 ... yes 34/1046 ... yes 201/1047 ... yes 126/1048 ... yes 34/1051 ... yes 154/1052 ... yes 34/1054 ... yes 138/1055 ... yes 37/1056 ... yes 62/1057 ... yes 13/1058 ... yes 105/1059 ... yes 136/1060 ... yes 176/1061 ... yes 136/1062 ... yes 161/1063 ... yes 19/1064 ... yes 4/1066 ... yes 208/1068 ... yes 88/1069 ... yes 28/1070 ... yes 160/1072 ... yes 209/1075 ... yes 161/1076 ... yes 161/1078 ... yes 139/1080 ... yes 199/1082 ... yes 28/1089 ... yes 212/1091 ... yes 212/1092 ... yes 212/1093 ... yes 210/1102 ... yes 200/1104 ... yes 212/1106 ... yes 212/1107 ... yes 176/1108 ... yes 106/1109 ... yes 211/1110 ... yes 211/1111 ... yes 24/1113 ... yes 28/1114 ... yes 106/1115 ... yes 176/1116 ... yes 138/1117 ... yes 138/1119 ... yes 215/1120 ... yes 220/1124 ... yes 106/1126 ... yes 24/1127 ... yes 209/1128 ... yes 34/1129 ... yes 161/1130 ... yes 34/1131 ... yes 190/1132 ... yes 105/1133 ... yes 106/1134 ... yes 139/1135 ... yes 176/1136 ... yes 34/1137 ... yes 210/1138 ... yes 210/1139 ... yes 113/1140 ... yes 211/1141 ... yes 161/1142 ... yes 161/1143 ... yes 161/1144 ... yes 161/1145 ... yes 209/1146 ... yes 161/1148 ... yes 225/1154 ... yes 115/1155 ... yes 113/1156 ... yes 225/1157 ... yes 113/1158 ... yes 145/1159 ... yes 259/1160 ... yes 275/1161 ... yes 211/1162 ... yes 233/1163 ... yes 105/1164 ... yes 225/1165 ... yes 44/1166 ... yes 106/1167 ... yes 265/1168 ... yes 237/1169 ... yes 28/1170 ... yes 162/1171 ... yes 179/1172 ... yes 24/1173 ... yes 241/1174 ... yes 88/1175 ... yes 200/1177 ... yes 24/1178 ... yes 34/1179 ... yes 244/1180 ... yes 246/1181 ... yes 246/1182 ... yes 28/1183 ... yes 28/1184 ... yes 176/1185 ... yes 247/1186 ... yes 179/1187 ... yes 248/1188 ... yes 28/1190 ... yes 161/1191 ... yes 105/1192 ... yes 136/1193 ... yes 249/1195 ... yes 250/1197 ... yes 251/1198 ... yes 96/1199 ... yes 1198/1202 ... yes 316/1204 ... yes 316/1205 ... yes 241/1208 ... yes 62/1209 ... yes 253/1210 ... yes 158/1213 ... yes 115/1214 ... yes 34/1215 ... yes 211/1216 ... yes 145/1219 ... yes 313/1220 ... yes 162/1221 ... yes 200/1223 ... yes 86/1224 ... yes 19/1225 ... yes 176/1226 ... yes 161/1228 ... yes 264/1232 ... yes 264/1233 ... yes 162/1235 ... yes 158/1237 ... yes 161/1240 ... yes 115/1242 ... yes 268/1243 ... yes 161/1244 ... yes 117/1245 ... yes 106/1246 ... yes 260/1247 ... yes 259/1248 ... yes 19/1249 ... yes 117/1250 ... yes 259/1251 ... yes 259/1252 ... yes 176/1253 ... yes 124/1255 ... yes 259/1256 ... yes 105/1257 ... yes 113/1258 ... yes 118/1260 ... yes 200/1264 ... yes 113/1265 ... yes 275/1267 ... yes 105/1269 ... yes 278/1270 ... yes 209/1271 ... yes 279/1272 ... yes 278/1273 ... yes 106/1274 ... yes 28/1275 ... yes 117/1276 ... yes 278/1278 ... yes 247/1279 ... yes 278/1280 ... yes 161/1281 ... yes 4/1282 ... yes 1198/1283 ... yes 162/1284 ... yes 161/1285 ... yes 161/1286 ... yes 161/1287 ... yes 145/1288 ... yes 136/1289 ... yes 281/1291 ... yes 282/1292 ... yes 65/1294 ... yes 283/1295 ... yes 284/1296 ... yes 282/1297 ... yes 284/1298 ... yes 282/1299 ... yes 28/1300 ... yes 162/1301 ... yes 286/1302 ... yes 288/1303 ... yes 19/1304 ... yes 19/1305 ... yes 136/1308 ... yes 288/1309 ... yes 176/1310 ... yes 163/1311 ... yes 225/1313 ... yes 145/1314 ... yes 291/1316 ... yes 184/1317 ... yes 249/1318 ... yes 106/1319 ... yes 285/1320 ... yes 288/1321 ... yes 284/1322 ... yes 210/1323 ... yes 136/1324 ... yes 45/1325 ... yes 24/1326 ... yes 163/1327 ... yes 136/1328 ... yes 145/1329 ... yes 249/1330 ... yes 204/1331 ... yes 161/1332 ... yes 301/1333 ... yes 211/1334 ... yes 24/1337 ... yes 28/1338 ... yes 305/1339 ... yes 305/1340 ... yes 4/1341 ... yes 303/1342 ... yes 306/1343 ... yes 138/1345 ... yes 161/1346 ... yes 291/1347 ... yes 303/1348 ... yes 225/1349 ... yes 138/1351 ... yes 184/1352 ... yes 113/1354 ... yes 316/1355 ... yes 106/1356 ... yes 317/1357 ... yes 24/1358 ... yes 204/1359 ... yes 106/1360 ... yes 113/1362 ... yes 24/1363 ... yes 329/1365 ... yes 4/1366 ... yes 324/1367 ... yes 324/1368 ... yes 1198/1369 ... yes 329/1370 ... yes 315/1371 ... yes 201/1373 ... yes 190/1374 ... yes 190/1375 ... yes 13/1377 ... yes 327/1378 ... yes 315/1379 ... yes 335/1380 ... yes 4/1381 ... yes 161/1382 ... yes 19/1383 ... yes 342/1384 ... yes 106/1385 ... yes 161/1386 ... yes 342/1387 ... yes 342/1388 ... yes 328/1389 ... yes 329/1391 ... yes 106/1392 ... yes 1198/1393 ... yes 136/1396 ... yes 339/1398 ... yes 339/1399 ... yes 260/1400 ... yes 1198/1401 ... yes 339/1402 ... yes 24/1403 ... yes 136/1404 ... yes 28/1405 ... yes 105/1406 ... yes 161/1407 ... yes 210/1408 ... yes 342/1409 ... yes 241/1410 ... yes 28/1412 ... yes 177/1413 ... yes 253/1414 ... yes 355/1415 ... yes 186/1418 ... yes 278/1419 ... yes 278/1420 ... yes 138/1421 ... yes 28/1422 ... yes 280/1423 ... yes 241/1424 ... yes 291/1425 ... yes 28/1426 ... yes 4/1427 ... yes 19/1428 ... yes 106/1430 ... yes 28/1431 ... yes 370/1433 ... yes 291/1435 ... yes 291/1436 ... yes 378/1439 ... yes 329/1440 ... yes 378/1442 ... yes 204/1443 ... yes 204/1444 ... yes 4/1445 ... yes 1198/1446 ... yes 398/1447 ... yes 398/1449 ... yes 345/1450 ... yes 317/1451 ... yes 398/1452 ... yes 4/1453 ... yes 4/1454 ... yes 4/1456 ... yes 4/1457 ... yes 136/1458 ... yes 4/1459 ... yes 4/1460 ... yes 4/1461 ... yes 106/1462 ... yes 398/1463 ... yes 278/1465 ... yes 278/1466 ... yes 136/1467 ... yes 398/1468 ... yes 1080/1470 ... yes 291/1472 ... yes 176/1473 ... yes 342/1476 ... yes 1099/1478 ... yes 138/1479 ... yes 83/1480 ... yes 24/1481 ... yes 163/1482 ... yes 106/1483 ... yes 398/1485 ... yes 126/1486 ... yes 190/1487 ... yes 1116/1489 ... yes 28/1490 ... yes 162/1491 ... yes 136/1492 ... yes 1132/1493 ... yes 1130/1495 ... yes 1131/1500 ... yes 1143/1501 ... yes 1133/1502 ... yes 1130/1504 ... yes 398/1505 ... yes 1116/1507 ... yes 1130/1508 ... yes 117/1509 ... yes 335/1511 ... yes 330/1512 ... yes 398/1513 ... yes 398/1516 ... yes 1167/1517 ... yes 398/1518 ... yes 1133/1519 ... yes 1219/1521 ... yes 190/1522 ... yes 1198/1523 ... yes 398/1524 ... yes 398/1525 ... yes 211/1526 ... yes 225/1527 ... yes 240/1528 ... yes 106/1529 ... yes 240/1530 ... yes 1087/1532 ... yes 4/1533 ... yes 162/1534 ... yes 398/1535 ... yes 190/1536 ... yes 1212/1537 ... yes 45/1538 ... yes 398/1539 ... yes 1219/1540 ... yes 1219/1541 ... yes 176/1542 ... yes 153/1543 ... yes 398/1544 ... yes 117/1546 ... yes 1247/1547 ... yes 1243/1548 ... yes 19/1549 ... yes 1247/1551 ... yes 398/1553 ... yes 1243/1554 ... yes 315/1555 ... yes 30/1556 ... yes 13/1557 ... yes 1130/1558 ... yes 329/1559 ... yes 1243/1560 ... yes 1247/1561 ... yes 398/1563 ... yes 106/1564 ... yes 328/1565 ... yes 1247/1566 ... yes 330/1567 ... yes 330/1569 ... yes 145/1570 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.7) Git user has default SSH configuration? ... yes Active users: ... 200 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
This is lib/gitlab/database/migration_helpers.rb (master) states this: validate_foreign_key() uses column argument. and uses inside foreign_key_exists? which has this multi column bug comment block. @See def foreign_key_exists?.
def validate_foreign_key(source, column, name: nil)
fk_name = name || concurrent_foreign_key_name(source, column)
unless foreign_key_exists?(source, name: fk_name)
raise missing_schema_object_message(source, "foreign key", fk_name)
end
disable_statement_timeout do
execute("ALTER TABLE #{source} VALIDATE CONSTRAINT #{fk_name};")
end
end
def foreign_key_exists?(source, target = nil, **options)
# This if block is necessary because foreign_key_exists? is called in down migrations that may execute before
# the postgres_foreign_keys view had necessary columns added.
# In that case, we revert to the previous behavior of this method.
# The behavior in the if block has a bug: it always returns false if the fk being checked has multiple columns.
# This can be removed after init_schema.rb passes 20221122210711_add_columns_to_postgres_foreign_keys.rb
# Tracking issue: https://gitlab.com/gitlab-org/gitlab/-/issues/386796
unless connection.column_exists?('postgres_foreign_keys', 'constrained_table_name')
return foreign_keys(source).any? do |foreign_key|
tables_match?(target.to_s, foreign_key.to_table.to_s) &&
options_match?(foreign_key.options, options)
end
end
fks = Gitlab::Database::PostgresForeignKey.by_constrained_table_name_or_identifier(source)
fks = fks.by_referenced_table_name(target) if target
fks = fks.by_name(options[:name]) if options[:name]
fks = fks.by_constrained_columns(options[:column]) if options[:column]
fks = fks.by_referenced_columns(options[:primary_key]) if options[:primary_key]
fks = fks.by_on_delete_action(options[:on_delete]) if options[:on_delete]
fks.exists?
endwhen /db/post_migrate/20230119095024_validate_fk_on_ci_builds_runner_session_partition_id_and_build_id.rb calls
COLUMNS = [:partition_id, :build_id]
def up
validate_foreign_key(TABLE_NAME, COLUMNS, name: FK_NAME)
end
validate_foreign_key with COLUMNS = [:partition_id, :build_id] this should lead to some error.