Branch and Tags missing after upgradation from GItLab 13.6.7 to 15.0.1
GitLab update performed from 13.6.7 to 15.0.1
In the current version, repos are migrated from legacy storage to hashed storage.Post upgrade below are observed.
- Latest commits are missing. Before previous update whatever updated is available.
- The branches created after previous upgrade are not available.
- Images uploaded in Issues are not available.
- Images uploaded in wiki are not available.
Updates performed in below manner to reach latest version
13.6.7 (Current Version)
13.9.2
13.12.15
14.0.12
14.5.0
14.9.0
14.10.4
15.0.1 (Latest Version)During 14.5.0 to 14.5.9 upgrade we get error.
Error 500 After LDAP Login in GUI
gitlab_rails['auto_migrate'] = true
gitlab-ctl reconfigure
DB Migration Failed.
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "batched_background_migration_job_transition_logs" does not exist
LINE 8: WHERE a.attrelid = '"batched_background_migration_job_trans...
Referred Link: https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
gitlab-rake db:migrate
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
Caused by:
PG::UndefinedTable: ERROR: relation "batched_background_migration_job_transition_logs" does not exist
LINE 8: WHERE a.attrelid = '"batched_background_migration_job_trans...
gitlab-ctl reconfigure
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
Refer: https://gitlab.com/gitlab-org/gitlab/-/issues/353927
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'
gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
[root@cm etc]# gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'
0
[root@cm etc]# gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
0
[root@cm etc]#
https://docs.gitlab.com/ee/update/index.html
gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'
https://docs.gitlab.com/ee/update/index.html
# Start the rails console
gitlab-rails c
# Execute the following in the rails console
scheduled_queue = Sidekiq::ScheduledSet.new
pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
[root@cm etc]# gitlab-rails c
--------------------------------------------------------------------------------
Ruby: ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
GitLab: 14.9.0 (77d81d7c756) FOSS
GitLab Shell: 13.24.0
PostgreSQL: 12.7
------------------------------------------------------------[ booted in 19.22s ]
Loading production environment (Rails 6.1.4.6)
irb(main):001:0> scheduled_queue = Sidekiq::ScheduledSet.new
=> #<Sidekiq::ScheduledSet:0x00007fc2f903cdc0 @name="schedule", @_size=0>
irb(main):002:0> pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.un
iq
=> []
irb(main):003:0> pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
=> []
irb(main):004:0>
=====================
# Start the rails console
gitlab-rails c
# Execute the following in the rails console
Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job|
puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}"
result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments)
puts "Result: #{result}"
end
[root@cm etc]# gitlab-rails c
--------------------------------------------------------------------------------
Ruby: ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
GitLab: 14.9.0 (77d81d7c756) FOSS
GitLab Shell: 13.24.0
PostgreSQL: 12.7
------------------------------------------------------------[ booted in 19.12s ]
Loading production environment (Rails 6.1.4.6)
irb(main):001:1* Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job|
irb(main):002:1* puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}"
irb(main):003:1* result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments)
irb(main):004:1* puts "Result: #{result}"
irb(main):005:0> end
=> nil
irb(main):006:0>
gitlab-rake db:migrate
gitlab-psql
select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
[root@cm etc]# gitlab-psql
psql (12.7)
Type "help" for help.
gitlabhq_production=# select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
job_class_name | table_name | column_name | job_arguments
---------------------------------------+---------------------+-------------+--------------------------------------------------------------------------
----------
CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata | id | [["build_id"], ["build_id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | push_event_payloads | event_id | [["event_id"], ["event_id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | ci_job_artifacts | id | [["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | taggings | id | [["id", "taggable_id"], ["id_convert_to_bigint", "taggable_id_convert_to_
bigint"]]
CopyColumnUsingBackgroundMigrationJob | ci_stages | id | [["id"], ["id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | events | id | [["id"], ["id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | ci_builds | id | [["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint
"]]
(7 rows)
gitlabhq_production=#
gitlab-rake db:migrate:up VERSION=20211123135255
[root@cm etc]# gitlab-rake db:migrate:up VERSION=20211123135255
== 20211123135255 CreateBatchedBackgroundMigrationJobTransitionLogs: migrating
-- execute("CREATE TABLE batched_background_migration_job_transition_logs (\n id bigserial NOT NULL,\n batched_background_migration_job_id bigint NOT NULL,\n created_at timestamp with time zone NOT NULL,\n updated_at timestamp with time zone NOT NULL,\n previous_status smallint NOT NULL,\n next_status smallint NOT NULL,\n exception_class text,\n exception_message text,\n CONSTRAINT check_50e580811a CHECK ((char_length(exception_message) <= 1000)),\n CONSTRAINT check_76e202c37a CHECK ((char_length(exception_class) <= 100)),\n PRIMARY KEY (id, created_at)\n) PARTITION BY RANGE (created_at);\n\nCREATE INDEX i_batched_background_migration_job_transition_logs_on_job_id\n ON batched_background_migration_job_transition_logs USING btree (batched_background_migration_job_id);\n\nALTER TABLE batched_background_migration_job_transition_logs ADD CONSTRAINT fk_rails_b7523a175b\n FOREIGN KEY (batched_background_migration_job_id) REFERENCES batched_background_migration_jobs(id) ON DELETE CASCADE;\n")
-> 0.0028s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_000000")
-> 0.0004s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_000000 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM (MINVALUE) TO ('2022-06-01')\n")
-> 0.5548s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202206")
-> 0.0005s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202206 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM ('2022-06-01') TO ('2022-07-01')\n")
-> 0.6750s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202207")
-> 0.0004s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202207 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM ('2022-07-01') TO ('2022-08-01')\n")
-> 0.5426s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202208")
-> 0.0005s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202208 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM ('2022-08-01') TO ('2022-09-01')\n")
-> 0.5090s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202209")
-> 0.0006s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202209 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM ('2022-09-01') TO ('2022-10-01')\n")
-> 0.6168s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202210")
-> 0.0005s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202210 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM ('2022-10-01') TO ('2022-11-01')\n")
-> 0.4915s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202211")
-> 0.0004s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202211 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM ('2022-11-01') TO ('2022-12-01')\n")
-> 0.5423s
-- table_exists?("gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202212")
-> 0.0005s
-- execute("CREATE TABLE gitlab_partitions_dynamic.batched_background_migration_job_transition_logs_202212 PARTITION OF batched_background_migration_job_transition_logs\nFOR VALUES FROM ('2022-12-01') TO ('2023-01-01')\n")
-> 0.5915s
== 20211123135255 CreateBatchedBackgroundMigrationJobTransitionLogs: migrated (4.5314s)
[root@cm etc]#
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
[root@cm etc]# gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
Done.
[root@cm etc]#
gitlab-ctl reconfigure
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
[root@cm etc]# gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
Done.
[root@cm etc]#
gitlab-rake db:migrate:up VERSION=20211123135255
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
gitlab.rb file modified as below
gitlab_rails['auto_migrate'] = false
gitlab-ctl reconfigure
gitlab-ctl restart
gitlab-ctl pg-upgrade
Upgrade logs and Postgresql steps attached.
*NOTE: *
Currently all data available under /var/opt/gitlab/git-data/repositories path.
In UI it referred @hashed path.
Read only repo state verified.
[root@cm backups]# gitlab-rails console
--------------------------------------------------------------------------------
Ruby: ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
GitLab: 13.6.7 (4d1fc8bce72) FOSS
GitLab Shell: 13.13.1
PostgreSQL: 11.10
--------------------------------------------------------------------------------
Loading production environment (Rails 6.0.3.3)
irb(main):001:0> projects = Project.where(repository_read_only: true)
=> #<ActiveRecord::Relation [#<Project id:444 cosmoss/cosmoss-mason>>, #<Project id:446 oss/google-dcn>>, #<Project id:214 manoharan/test-automat...
irb(main):002:1* projects.each do |p|
irb(main):003:1* p.update!(repository_read_only:nil)
irb(main):004:0> end
=> [#<Project id:444 cosmoss/cosmoss-mason>>, #<Project id:446 oss/google-dcn>>, #<Project id:214 manoharan/test-automation>>, #<Project id:213 sakthi/point>>]
irb(main):005:0>
[root@cm home]# gitlab-rake gitlab:storage:migrate_to_hashed
Enqueuing migration of 46 projects in batches of 200. Done!
[root@cm home]#
Executed below script to convert from Legacy to Hashed
# Find all projects that GitLab thinks is in legacy storage
Project.without_storage_feature(:repository).find_each(batch_size: 10) do |p|
# Generate the hashed path
hash_path = Storage::Hashed.new(p).disk_path
storage = Gitlab.config.repositories.storages[p.repository.shard]
# Enable access
Gitlab::GitalyClient::StorageSettings.allow_disk_access do
# Put a wildcard on the end so we find .git, .wiki.git and .design.git
full_hashed_path = File.expand_path(File.join(storage.legacy_disk_path, hash_path + '*'))
# If any repos already exist in hashed storage, complete the migration
if(!Dir.glob(full_hashed_path).empty?)
puts "Fixing ID:#{p.id} #{p.full_name} - #{full_hashed_path}"
# Set the repo to read only if it isn't already
if(!p.repository_read_only)
p.set_repository_read_only!
end
p.storage_version = ::Project::HASHED_STORAGE_FEATURES[:repository]
p.reload_repository!
p.write_repository_config
p.track_project_repository
p.repository_read_only = false
p.save!(validate: false)
end
end
end
Logs attached. prod_postgresql_update_09062022.txt
Edited by Manoharan M