Skip to content
  • Nick Nguyen @nhxnguyen ·

    @anton Should GitLab then gets into a weird situation where it thinks the repository files exist in legacy storage, but they actually exist in legacy storage. be they actually exist in hashed storage?

  • Anton Smith @anton ·

    @nhxnguyen Thanks for the catch there, I have updated the description :)

  • Nick Nguyen @nhxnguyen ·

    Sure thing. Thank you for providing a workaround with this snippet! 😄

  • Bergis Hache @agentsmith80 ·

    Thank you so much @anton for this fix!!

  • Cody West @cody ·

    For those that can't wget the script, you can just run it directly in the rails console

    sudo gitlab-rails console
# then copy the code from https://gitlab.com/snippets/2039252/raw
# paste the full script once you see the prompt
>
  • Kopax Anderson @d-kopriwa ·

    I have this:

    root@private:/# gitlab-rake gitlab:storage:list_legacy_projects
* Found 6 projects using Legacy Storage
  - dka-veryold/HTML5-Game-Multiplayer-presentation (id: 23)
  - dka-veryold/reveal (id: 24)
  - dka-veryold/sanslesmains-api (id: 25)
  - dka-veryold/sanslesmains-lab (id: 27)
  - dka-veryold/monitoring-countdown (id: 33)
  - kopax-old/seafile (id: 37)
root@private:/# gitlab-rake gitlab:storage:list_legacy_attachments
* Found 0 attachments using Legacy Storage
root@private:/# gitlab-rake gitlab:storage:migrate_to_hashed
Enqueuing migration of 6 projects in batches of 200. Done!
root@private:/# gitlab-rake gitlab:storage:list_legacy_projects
* Found 6 projects using Legacy Storage
  - dka-veryold/HTML5-Game-Multiplayer-presentation (id: 23)
  - dka-veryold/reveal (id: 24)
  - dka-veryold/sanslesmains-api (id: 25)
  - dka-veryold/sanslesmains-lab (id: 27)
  - dka-veryold/monitoring-countdown (id: 33)
  - kopax-old/seafile (id: 37)
root@private:/#
    1. Run the sudo gitlab-rake gitlab:storage:migrate_to_hashed command, to ensure that project attachments are also migrated.

    How do your run the migration?

  • Kopax Anderson @d-kopriwa ·

    It seems that the migration script does not work, if I use ID_FROM=54 ID_TO=54, it says the ID does not exist. How can I recover my gitlab? please

  • Kopax Anderson @d-kopriwa ·

    See full report here. Completely stuck migrating and that's my latest backup after the OVH incident Strasbourg BC got entirely burned.

  • Kopax Anderson @d-kopriwa ·

    I have this on a second node :

    root@module:/var/opt/gitlab/backups# wget -O /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2039252/raw
--2021-03-29 23:12:45--  https://gitlab.com/snippets/2039252/raw
Resolving gitlab.com (gitlab.com)... 172.65.251.78, 2606:4700:90:0:f22e:fbec:5bed:a9b9
Connecting to gitlab.com (gitlab.com)|172.65.251.78|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘/tmp/fix-legacy-hashed-storage-migration.rb’

/tmp/fix-legacy-hashed-storage-migration.rb              [ <=>                                                                                                                  ]   1.04K  --.-KB/s    in 0s      

2021-03-29 23:12:45 (6.66 MB/s) - ‘/tmp/fix-legacy-hashed-storage-migration.rb’ saved [1061]

root@module:/var/opt/gitlab/backups# gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb

Fixing ID:6 styled-components / redstar-squeleton-old - /var/opt/gitlab/git-data/repositories/@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683*
Traceback (most recent call last):
	58: from bin/rails:4:in `<main>'
	57: from bin/rails:4:in `require'
	56: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands.rb:18:in `<top (required)>'
	55: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command.rb:46:in `invoke'
	54: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command/base.rb:69:in `perform'
	53: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	52: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	51: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	50: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/runner/runner_command.rb:42:in `perform'
	49: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/runner/runner_command.rb:42:in `load'
	48: from /tmp/fix-legacy-hashed-storage-migration.rb:2:in `<top (required)>'
	47: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:69:in `find_each'
	46: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:135:in `find_in_batches'
	45: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:222:in `in_batches'
	44: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:222:in `loop'
	43: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
	42: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
	41: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:70:in `block in find_each'
	40: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:70:in `each'
	39: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
	38: from /tmp/fix-legacy-hashed-storage-migration.rb:27:in `block in <top (required)>'
	37: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/suppressor.rb:48:in `save!'
	36: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `save!'
	35: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
	34: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:212:in `transaction'
	33: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
	32: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
	31: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
	30: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
	29: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
	28: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
	27: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
	26: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
	25: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
	24: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
	23: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `block in save!'
	22: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/validations.rb:53:in `save!'
	21: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/persistence.rb:503:in `save!'
	20: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/timestamp.rb:128:in `create_or_update'
	19: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/callbacks.rb:327:in `create_or_update'
	18: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
	17: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:134:in `run_callbacks'
	16: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `invoke_before'
	15: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `each'
	14: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `block in invoke_before'
	13: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:201:in `block in halting'
	12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:604:in `block in default_terminator'
	11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:604:in `catch'
	10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:605:in `block (2 levels) in default_terminator'
	 9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting'
	 8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'
	 7: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
	 6: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
	 5: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
	 4: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
	 3: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
	 2: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
	 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final': OpenSSL::Cipher::CipherError

    How can I migrate my repository and must this be fixed ? @cody

  • Kopax Anderson @d-kopriwa ·

    @cody, this is what I have after restoring from another backup and running the migration script after upgrading to 13.9.5

    Traceback (most recent call last):
	21: from bin/rails:4:in `<main>'
	20: from bin/rails:4:in `require'
	19: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/commands.rb:18:in `<top (required)>'
	18: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/command.rb:46:in `invoke'
	17: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/command/base.rb:69:in `perform'
	16: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
	15: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
	14: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
	13: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/commands/runner/runner_command.rb:42:in `perform'
	12: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/commands/runner/runner_command.rb:42:in `load'
	11: from /tmp/fix-legacy-hashed-storage-migration.rb:2:in `<top (required)>'
	10: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
	 9: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
	 8: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
	 7: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
	 6: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
	 5: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
	 4: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
	 3: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
	 2: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
	 1: from /tmp/fix-legacy-hashed-storage-migration.rb:9:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/storage_settings.rb:64:in `legacy_disk_path': git disk access denied (Gitlab::GitalyClient::StorageSettings::DirectPathAccessError)

    What are the other way to migrate those repos?

  • Kopax Anderson @d-kopriwa ·

    After some debugging and running the rb script in console directly, I was able to log this error message:

    #<Gitlab::GitalyClient::StorageSettings:0x00007f6bfa803a78 @legacy_disk_path="/var/opt/gitlab/git-data/repositories", @hash={"path"=>Gitlab::GitalyClient::StorageSettings::Deprecated, "gitaly_address"=>"unix:/var/opt/gitlab/gitaly/gitaly.socket"}>
Traceback (most recent call last):
        3: from (irb):2
        2: from (irb):14:in `block in irb_binding'
        1: from lib/gitlab/gitaly_client/storage_settings.rb:64:in `legacy_disk_path'
Gitlab::GitalyClient::StorageSettings::DirectPathAccessError (git disk access denied)

    So I presume this could be fixed somehow quite easily...

  • Cody West @cody ·

    @d-kopriwa it looks like that's a simple permissions problem on git disk access. Make sure that the Gitaly service user has full permissions to /var/opt/gitlab/git-data/repositories.

  • Will Chandler @wchandler ·

    I'm running into the same error on my 13.10 instance when trying this snippet. Looks like there's a feature flag was enabled that denies access by default now.

    @d-kopriwa we should be able to work around this by wrapping the request in a Gitlab::GitalyClient::StorageSettings.allow_disk_access do block:

    # Find all projects that GitLab thinks is in legacy storage
Project.without_storage_feature(:repository).find_each(batch_size: 10) do |p|

    # Generate the hashed path
    hash_path = Storage::Hashed.new(p).disk_path
    storage = Gitlab.config.repositories.storages[p.repository.shard]

    # Enable access
    Gitlab::GitalyClient::StorageSettings.allow_disk_access do

        # Put a wildcard on the end so we find .git, .wiki.git and .design.git
        full_hashed_path = File.expand_path(File.join(storage.legacy_disk_path, hash_path + '*'))

        # If any repos already exist in hashed storage, complete the migration
        if(!Dir.glob(full_hashed_path).empty?)
            puts "Fixing ID:#{p.id} #{p.full_name} - #{full_hashed_path}"
        
            # Set the repo to read only if it isn't already
            if(!p.repository_read_only)
                p.set_repository_read_only!
            end

            p.storage_version = ::Project::HASHED_STORAGE_FEATURES[:repository]

            p.reload_repository!
            p.write_repository_config
            p.track_project_repository

            p.repository_read_only = false
            p.save!(validate: false)
        end
    end
end
  • Will Chandler @wchandler ·

    @anton I think you should have edit access to this snippet. Can you update it to include the allow_disk_access block as in https://gitlab.com/-/snippets/2039252#note_553815038?

  • Kopax Anderson @d-kopriwa ·

    Hi @cody and thanks for your reply and help,

    @d-kopriwa it looks like that's a simple permissions problem on git disk access. Make sure that the Gitaly service user has full permissions to /var/opt/gitlab/git-data/repositories.

    I already checked with another instance of gitlab and permission are set accordingly, I believe the issue was with the script and @anton just posted a new version of it.

    This is the whole line I have run which use the fixed script for allow_disk_access error:

    wget -O /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2106558/raw
gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb; gitlab-rake gitlab:storage:migrate_to_hashed; gitlab-rake gitlab:storage:list_legacy_projects; gitlab-rake gitlab:storage:list_legacy_attachments

    It doesn't crash anymore with the fix, however, I still have the following output not migrated:

    root@module:/# wget -O  /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2106558/raw && \
  gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb && \
  gitlab-rake gitlab:storage:migrate_to_hashed && \
  gitlab-rake gitlab:storage:list_legacy_projects && \
  gitlab-rake gitlab:storage:list_legacy_attachments

Fixing ID:112 bootstrap-styled / bootstrap-styled-saga - /var/opt/gitlab/git-data/repositories/@hashed/b1/55/b1556dea32e9d0cdbfed038fd7787275775ea40939c146a64e205bcb349ad02f*
Fixing ID:114 dev-tools / rollup-umd-scripts - /var/opt/gitlab/git-data/repositories/@hashed/9f/1f/9f1f9dce319c4700ef28ec8c53bd3cc8e6abe64c68385479ab89215806a5bdd6*
Fixing ID:116 dev-tools / rollup-documentation - /var/opt/gitlab/git-data/repositories/@hashed/e5/b8/e5b861a6d8a966dfca7e7341cd3eb6be9901688d547a72ebed0b1f5e14f3d08d*
Fixing ID:122 bootstrap-styled / bootstrap-styled-theme - /var/opt/gitlab/git-data/repositories/@hashed/1b/e0/1be00341082e25c4e251ca6713e767f7131a2823b0052caf9c9b006ec512f6cb*
Fixing ID:129 dimitri.kopriwa / gitlab-ci-cli - /var/opt/gitlab/git-data/repositories/@hashed/65/66/6566230e3a3ce3774c1bbc7c18b590ae0f457bbcd511e90e3e7dca2a02e7addc*
Fixing ID:165 rollup-umd / rollup-umd-ci-docker - /var/opt/gitlab/git-data/repositories/@hashed/bc/52/bc52dd634277c4a34a2d6210994a9a5e2ab6d33bb4a3a8963410e00ca6c15a02*
Fixing ID:177 kopaxgroup / workstation - /var/opt/gitlab/git-data/repositories/@hashed/8c/d2/8cd2510271575d8430c05368315a87b9c4784c7389a47496080c1e615a2a00b6*
Fixing ID:179 bootstrap-styled / v4 - /var/opt/gitlab/git-data/repositories/@hashed/30/68/3068430da9e4b7a674184035643d9e19af3dc7483e31cc03b35f75268401df77*
Fixing ID:181 bootstrap-styled / utils - /var/opt/gitlab/git-data/repositories/@hashed/58/08/580811fa95269f3ecd4f22d176e079d36093573680b6ef66fa341e687a15b5da*
Fixing ID:182 bootstrap-styled / redux - /var/opt/gitlab/git-data/repositories/@hashed/bf/a7/bfa7634640c53da7cb5e9c39031128c4e583399f936896f27f999f1d58d7b37e*
Fixing ID:183 deploy-tools / ssh-deploy - /var/opt/gitlab/git-data/repositories/@hashed/b8/ae/b8aed072d29403ece56ae9641638ddd50d420f950bde0eefc092ee8879554141*
Fixing ID:185 bootstrap-styled / provider - /var/opt/gitlab/git-data/repositories/@hashed/61/a2/61a229bae1e90331edd986b6bbbe617f7035de88a5bf7c018c3add6c762a6e8d*
Fixing ID:186 bootstrap-styled / css-mixins - /var/opt/gitlab/git-data/repositories/@hashed/28/11/2811745d7b8d8874f6e653d176cefdd19e05e920ce389b9b7e83e5b2dfa546c7*
Fixing ID:187 bootstrap-styled / toggle - /var/opt/gitlab/git-data/repositories/@hashed/38/b2/38b2d03f3256502b1e9db02b2d12aa27a46033ffe6d8c0ef0f2cf6b1530be9d8*
Fixing ID:188 bootstrap-styled / css-utils - /var/opt/gitlab/git-data/repositories/@hashed/d6/06/d6061bbee6cf13bd73765faaea7cdd0af1323e4b125342ac346047f7c4bda1fc*
Fixing ID:190 dimitri.kopriwa / samba - /var/opt/gitlab/git-data/repositories/@hashed/23/97/2397346b45823e070f6fc72ac94c0a999d234c472479f0e26b30cdf5942db854*
Fixing ID:192 yeutech / www-assets - /var/opt/gitlab/git-data/repositories/@hashed/eb/3b/eb3be230bbd2844b1f5d8f2e4fab9ffba8ab22cfeeb69c4c1361993ba4f377b9*
Fixing ID:194 rollup-umd / documentation-cli - /var/opt/gitlab/git-data/repositories/@hashed/75/59/7559ca4a957c8c82ba04781cd66a68d6022229fca0e8e88d8e487c96ee4446d0*
Fixing ID:195 bootstrap-styled / rsg-components - /var/opt/gitlab/git-data/repositories/@hashed/1d/fa/1dfacb2ea5a03e0a915999e03b5a56196f1b1664d2f768d1b7eff60ac059789d*
Fixing ID:196 rollup-umd / documentation - /var/opt/gitlab/git-data/repositories/@hashed/b4/bb/b4bbe448fde336bb6a7d7d765f36d3327c772b845e7b54c8282aa08c9775ddd7*
Fixing ID:197 bootstrap-styled / documentation - /var/opt/gitlab/git-data/repositories/@hashed/8b/cb/8bcbb4c131df56f7c79066016241cc4bdf4e58db55c4f674e88b22365bd2e2ad*
Fixing ID:198 yeutech / documentation - /var/opt/gitlab/git-data/repositories/@hashed/a4/e0/a4e00d7e6aa82111575438c5e5d3e63269d4c475c718b2389f6d02932c47f8a6*
Fixing ID:199 bootstrap-styled / color - /var/opt/gitlab/git-data/repositories/@hashed/5a/39/5a39cadd1b007093db50744797c7a04a34f73b35ed444704206705b02597d6fd*
Fixing ID:201 yeutech / react-router-dom-utils - /var/opt/gitlab/git-data/repositories/@hashed/43/97/43974ed74066b207c30ffd0fed5146762e6c60745ac977004bc14507c7c42b50*
Fixing ID:202 yeutech / ra-language-intl - /var/opt/gitlab/git-data/repositories/@hashed/c1/7e/c17edaae86e4016a583e098582f6dbf3eccade8ef83747df9ba617ded9d31309*
Fixing ID:203 bootstrap-styled / ra-ui - /var/opt/gitlab/git-data/repositories/@hashed/46/21/4621c1d55fa4e86ce0dae4288302641baac86dd53f76227c892df9d300682d41*
Fixing ID:204 yeutech / react-admin-intl - /var/opt/gitlab/git-data/repositories/@hashed/fc/56/fc56dbc6d4652b315b86b71c8d688c1ccdea9c5f1fd07763d2659fde2e2fc49a*
Fixing ID:206 yeutech / react-intl-polyfill - /var/opt/gitlab/git-data/repositories/@hashed/5c/f4/5cf4e26bd3d87da5e03f80a43a64f1220a1f4ba9e1d6348caea83c06353c3f39*
Fixing ID:207 dimitri.kopriwa / example-tue - /var/opt/gitlab/git-data/repositories/@hashed/96/80/968076be2e38cf897d4d6cea3faca9c037e1a4e3b4b7744fb2533e07751bd30a*
Fixing ID:208 yeutech / bitcoin-currency-converter - /var/opt/gitlab/git-data/repositories/@hashed/8d/f6/8df66f64b57424391d363fd6b811fed3c430c77597da265025728bd637bad804*
Fixing ID:209 yeutech / test-polyfill - /var/opt/gitlab/git-data/repositories/@hashed/83/f8/83f814f7a92e365cbd79f9addceed185761a8d38a06a2d4350bb1fe4b7632b34*
Fixing ID:211 kopaxgroup-api / user-management - /var/opt/gitlab/git-data/repositories/@hashed/09/34/093434a3ee9e0a010bb2c2aae06c2614dd24894062a1caf26718a01e175569b8*
Fixing ID:214 kopaxgroup / documentation - /var/opt/gitlab/git-data/repositories/@hashed/80/2b/802b906a18591ead8a6dd809b262ace4c65c16e89764c40ae326cfcff811e10c*
Fixing ID:215 kopaxgroup-api / site-service - /var/opt/gitlab/git-data/repositories/@hashed/d8/65/d86580a57f7bf542e85202283cb845953c9d28f80a8e651db08b2fc0b2d6a731*
Fixing ID:217 kopaxgroup-api / um-users - /var/opt/gitlab/git-data/repositories/@hashed/16/ba/16badfc6202cb3f8889e0f2779b19218af4cbb736e56acadce8148aba9a7a9f8*
Fixing ID:218 bootstrap-styled / ra-ui-extends - /var/opt/gitlab/git-data/repositories/@hashed/59/66/5966abd0cbfc86f98a186531b2b4ee5f6e910120ce13222f98207203dfc9a9a2*
Fixing ID:219 yeutech / webpack-utils - /var/opt/gitlab/git-data/repositories/@hashed/31/4f/314f04b30f62e0056bd059354a5536fb2e302107eed143b5fa2aa0bbba07f608*
Fixing ID:220 yeutech / app-context - /var/opt/gitlab/git-data/repositories/@hashed/36/79/36790ecd55c2030dc553685bef719df653f413a20cdad1bfd1dc934c76686ddd*
Enqueuing migration of 16 projects in batches of 200. Done!
* Found 16 projects using Legacy Storage
  - rollup-umd/rollup-umd-documentation-cli (id: 148)
    rollup-umd/rollup-umd-documentation-cli
  - rollup-umd/rollup-umd-ci-test (id: 150)
    rollup-umd/rollup-umd-ci-test
  - rollup-umd/rollup-umd-ci-release (id: 151)
    rollup-umd/rollup-umd-ci-release
  - rollup-umd/rollup-umd-ci-deploy (id: 152)
    rollup-umd/rollup-umd-ci-deploy
  - rollup-umd/rollup-umd-ci-release-transitive (id: 153)
    rollup-umd/rollup-umd-ci-release-transitive
  - yeutech/image-common (id: 161)
    yeutech/image-common
  - kopaxgroup/email-cli (id: 162)
    kopaxgroup/email-cli
  - basic/email-templates (id: 167)
    basic/email-templates
  - dka/email-templates-yeutech-cli (id: 169)
    dka/email-templates-yeutech-cli
  - yeutech/email-templates-yeutech (id: 171)
    yeutech/email-templates-yeutech
  - dev-tools/react-html-email-cli (id: 173)
    dev-tools/react-html-email-cli
  - yeutech/email-campaign-yeutech-cli (id: 174)
    yeutech/email-campaign-yeutech-cli
  - basic/language-provider-factory (id: 176)
    basic/language-provider-factory
  - yeutech/wallpapers-yeutech (id: 178)
    yeutech/wallpapers-yeutech
  - bootstrap-styled/v4.yeutech.com (id: 180)
    bootstrap-styled/v4.yeutech.com
  - deploy-tools/git-url-tweak (id: 184)
    deploy-tools/git-url-tweak
* Found 0 attachments using Legacy Storage

    Any clue how I can fix all of them? I'd like to be able to keep GitLab long term up to date.

    Thanks !

    Edited by Kopax Anderson
  • Anton Smith @anton ·

    @wchandler Sorry I was on OOO. I have updated the script in the snippet with your changes.

    @d-kopriwa It looks the snippet fixed alot of your projects. I assume that the fixed projects are visible in GitLab now? For these projects that are still in legacy storage:

      - rollup-umd/rollup-umd-documentation-cli (id: 148)
  - rollup-umd/rollup-umd-ci-test (id: 150)
  - rollup-umd/rollup-umd-ci-release (id: 151)
  - rollup-umd/rollup-umd-ci-deploy (id: 152)
  - rollup-umd/rollup-umd-ci-release-transitive (id: 153)
  - yeutech/image-common (id: 161)
  - kopaxgroup/email-cli (id: 162)
  - basic/email-templates (id: 167)
  - dka/email-templates-yeutech-cli (id: 169)
  - yeutech/email-templates-yeutech (id: 171)
  - dev-tools/react-html-email-cli (id: 173)
  - yeutech/email-campaign-yeutech-cli (id: 174)
  - basic/language-provider-factory (id: 176)
  - yeutech/wallpapers-yeutech (id: 178)
  - bootstrap-styled/v4.yeutech.com (id: 180)
  - deploy-tools/git-url-tweak (id: 184)
    1. Can you view these projects/files in the GitLab UI? Or does it show an empty repository?
    2. The migration might be failing for another reason. Can you run sudo gitlab-rake gitlab:storage:migrate_to_hashed to try and migrate these projects again? Afterwards, please wait 10 minutes to give the async jobs time to perform the migration. Then:
      • Run gitlab-rake gitlab:storage:list_legacy_projects to check if this fixed the problem (if it did, the projects won't appear in the list)
      • If not, let's look at the logs grep "HashedStorage::ProjectMigrateWorker" /var/log/gitlab/sidekiq/current (please attach any logs as files rather than copy/pasting the contents into your message). Chances are there is an error/stacktrace in here.
  • PBillodeau @PBillodeau ·

    I appreciate your work creating this script! It fixed my inability to migrate the repos to hashed storage, but caused another issue -- any commit made in the last ~10 months disappeared from the repository fixed. 10 months might correspond the last time I made a major update (12 to 13).

    Has anyone else run into this problem, or have any idea why it happened? Any help is appreciated!

    Additional info:

    • Gitlab 13.12.3 (Omnibus install)
    • Gitlab is running behind Apache reverse proxy, NGINX is disabled, otherwise configuration is pretty much just the defaults
  • PBillodeau @PBillodeau ·

    Huge disclaimer: Do not use this script unless you are 100% sure, and have a backup of your data ready to go! Unless you're in the exact same situation as I was, It has a high probability of destroying data.

    In continuation of my last post, in my case, it seems that ~10 months ago, Gitlab moved the repositories in the file system from legacy to hashed, but didn't identify that it had done so, which caused subsequent commits to be store in the legacy path. This all happened unbeknownst to me. Then, once I finally ran into an issue (trying to upgrade to Gitlab 14) I found this script. Running the script fixed the issue where Gitlab didn't identify that the repository was using hashed storage, but it didn't fix the subsequent commits stored in the legacy path because they shouldn't have been there anymore!

    I made a change to the original script that moves all of the files from the legacy path to the hashed path. In my case, it works because the legacy path contained the entire history of the repo. I'm not very familiar with Ruby, so it uses a system call instead of doing it the Ruby way, but hopefully it will at least help someone identify their problem.

    # Find all projects that GitLab thinks is in legacy storage
Project.without_storage_feature(:repository).find_each(batch_size: 10) do |p|

    # Generate the hashed path
    hash_path = Storage::Hashed.new(p).disk_path
    storage = Gitlab.config.repositories.storages[p.repository.shard]

    # Enable access
    Gitlab::GitalyClient::StorageSettings.allow_disk_access do

        # Put a wildcard on the end so we find .git, .wiki.git and .design.git
        full_hashed_path = File.expand_path(File.join(storage.legacy_disk_path, hash_path + '*'))

        # If any repos already exist in hashed storage, complete the migration
        if(!Dir.glob(full_hashed_path).empty?)
            puts "Fixing ID:#{p.id} #{p.full_name} - #{full_hashed_path}"

            # Set the repo to read only if it isn't already
            if(!p.repository_read_only)
                p.set_repository_read_only!
            end

            # Copy over git files from legacy path to hashed path
            if(!Dir.glob(File.expand_path(File.join(storage.legacy_disk_path, p.disk_path + '.git'))).empty?)
                system("cp -rf #{File.expand_path(File.join(storage.legacy_disk_path, p.disk_path + '.git', '*'))} #{File.expand_path(File.join(storage.legacy_disk_path, hash_path + '.git'))}")
            end

            p.storage_version = ::Project::HASHED_STORAGE_FEATURES[:repository]

            p.reload_repository!
            p.write_repository_config
            p.track_project_repository

            p.repository_read_only = false
            p.save!(validate: false)
        end
    end
end
  • Alexis Lefebvre 🐼 @alexislefebvre ·

    @PBillodeau Thanks! I had the same issue: the script on the top of the page made some commits disappear. It looks like your version of the script fixed the issue. I restored the last backup, then ran your script, and I was able to migrate to GitLab 14.

    Edited by Alexis Lefebvre
  • Adam Beauchamp @adrock20 ·

    More recent versions of gitlab that include b0e2977c3dd5d837216a38747df853cece58c28b require write_repository_config to change to set_full_path in the above script.

    Above change allowed successful running on Gitlab 15.4.3

    Edited by Adam Beauchamp
  • Martha Simons @marthasimons ·

    The visuals, diagrams, and screenshots used are a great👍 addition. They visually reinforce the concepts and provide a visual aid that complements the textual explanations perfectly.

    One thing I found particularly impressive was the troubleshooting section. It covered common issues that beginners often face and provided effective solutions. This attention to detail and consideration for the readers' potential challenges sets this page apart from others I've come across.

    Overall, this page has given me a solid foundation in Git and boosted my confidence in using version control. I highly recommend it to anyone starting their Git journey or looking to reinforce their understanding. Kudos to the author for creating such an invaluable resource!

  • Anton Vasilev @tonycode ·

    For those who will face this issue and:

    • see a text RepositoryExists in the logs after running gitlab-rake gitlab:storage:migrate_to_hashed
    • and/or got an error with cause OpenSSL::Cipher::CipherError running original @anton's script (fix-legacy-hashed-storage-migration.rb) or allow_disk_access-modified @PBillodeau's version (https://gitlab.com/-/snippets/2039252#note_629079354)

    here's a tip: "reset runner registration tokens"

    on Gitlab 13.0.* this is done by:

    gitlab-rails dbconsole

-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;

    my case:

    1. I've got a Gitlab-12.10.14 backup made in 2021 and I need to restore it and upgrade to the latest Gitlab-CE version
    2. backup was restored in Gitlab-12.10.14 (down sidekiq/unicorn, gitlab-backup restore BACKUP=..., manually copy /srv/gitlab/config secrets dir, restart, then all sanity-checks)
    3. upgraded to Gitlab-13.0.14 (docker stop gitlab && docker rm gitlab && docker run ... 13.0.14 then check that everything is ok)
    4. ran the code from above "reset runner reg-tokens"
    5. ran gitlab-rake gitlab:storage:migrate_to_hashed
    6. gave previous command a couple of minutes, then checked gitlab-rake gitlab:storage:list_legacy_projects and gitlab-rake gitlab:storage:list_legacy_attachments - none of legacy projects was left. Btw checked that gitlab-rake gitlab:check SANITIZE=true shows no errors/warnings. DONE

    without item-4 - I've got about 1/3 of projects kept in legacy state after item-5, and fix-legacy-*.rb scripts were failing with OpenSSL::Cipher::CipherError

    the idea of adding item-4 was originally proposed there - https://knowledge.koba.jp/open.knowledge/view/14 - many thanks to this guy

    Hope this helps somebody,

    UPD: found same fix here - https://forum.gitlab.com/t/upgrade-to-gitlab-13-4-0-b0481767fe4-killed-all-repositories/43038/39

    Edited by Anton Vasilev
  • Brain Tyler @BrainTyler ·

    Hey,

    Instead of using this it's better that you will use writing websites (Like Essay Basics, EssayKitchen, EduJungles that will help you to get best content without plagiarism. Their is no surety in AI or Bot to get 100% orignal content.

0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment