• @anton Should GitLab then gets into a weird situation where it thinks the repository files exist in legacy storage, but they actually exist in legacy storage. be they actually exist in hashed storage?

  • @nhxnguyen Thanks for the catch there, I have updated the description :)

  • Sure thing. Thank you for providing a workaround with this snippet! 😄

  • Thank you so much @anton for this fix!!

  • For those that can't wget the script, you can just run it directly in the rails console

    sudo gitlab-rails console
    # then copy the code from https://gitlab.com/snippets/2039252/raw
    # paste the full script once you see the prompt
    > 
  • I have this:

    root@private:/# gitlab-rake gitlab:storage:list_legacy_projects
    * Found 6 projects using Legacy Storage
      - dka-veryold/HTML5-Game-Multiplayer-presentation (id: 23)
      - dka-veryold/reveal (id: 24)
      - dka-veryold/sanslesmains-api (id: 25)
      - dka-veryold/sanslesmains-lab (id: 27)
      - dka-veryold/monitoring-countdown (id: 33)
      - kopax-old/seafile (id: 37)
    root@private:/# gitlab-rake gitlab:storage:list_legacy_attachments
    * Found 0 attachments using Legacy Storage
    root@private:/# gitlab-rake gitlab:storage:migrate_to_hashed
    Enqueuing migration of 6 projects in batches of 200. Done!
    root@private:/# gitlab-rake gitlab:storage:list_legacy_projects
    * Found 6 projects using Legacy Storage
      - dka-veryold/HTML5-Game-Multiplayer-presentation (id: 23)
      - dka-veryold/reveal (id: 24)
      - dka-veryold/sanslesmains-api (id: 25)
      - dka-veryold/sanslesmains-lab (id: 27)
      - dka-veryold/monitoring-countdown (id: 33)
      - kopax-old/seafile (id: 37)
    root@private:/# 
    1. Run the sudo gitlab-rake gitlab:storage:migrate_to_hashed command, to ensure that project attachments are also migrated.

    How do your run the migration?

  • It seems that the migration script does not work, if I use ID_FROM=54 ID_TO=54, it says the ID does not exist. How can I recover my gitlab? please

  • See full report here. Completely stuck migrating and that's my latest backup after the OVH incident Strasbourg BC got entirely burned.

  • I have this on a second node :

    root@module:/var/opt/gitlab/backups# wget -O /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2039252/raw
    --2021-03-29 23:12:45--  https://gitlab.com/snippets/2039252/raw
    Resolving gitlab.com (gitlab.com)... 172.65.251.78, 2606:4700:90:0:f22e:fbec:5bed:a9b9
    Connecting to gitlab.com (gitlab.com)|172.65.251.78|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: unspecified [text/plain]
    Saving to: ‘/tmp/fix-legacy-hashed-storage-migration.rb’
    
    /tmp/fix-legacy-hashed-storage-migration.rb              [ <=>                                                                                                                  ]   1.04K  --.-KB/s    in 0s      
    
    2021-03-29 23:12:45 (6.66 MB/s) - ‘/tmp/fix-legacy-hashed-storage-migration.rb’ saved [1061]
    
    root@module:/var/opt/gitlab/backups# gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb
    
    Fixing ID:6 styled-components / redstar-squeleton-old - /var/opt/gitlab/git-data/repositories/@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683*
    Traceback (most recent call last):
    	58: from bin/rails:4:in `<main>'
    	57: from bin/rails:4:in `require'
    	56: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands.rb:18:in `<top (required)>'
    	55: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command.rb:46:in `invoke'
    	54: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command/base.rb:69:in `perform'
    	53: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    	52: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    	51: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    	50: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/runner/runner_command.rb:42:in `perform'
    	49: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/runner/runner_command.rb:42:in `load'
    	48: from /tmp/fix-legacy-hashed-storage-migration.rb:2:in `<top (required)>'
    	47: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:69:in `find_each'
    	46: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:135:in `find_in_batches'
    	45: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:222:in `in_batches'
    	44: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:222:in `loop'
    	43: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
    	42: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
    	41: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:70:in `block in find_each'
    	40: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:70:in `each'
    	39: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
    	38: from /tmp/fix-legacy-hashed-storage-migration.rb:27:in `block in <top (required)>'
    	37: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/suppressor.rb:48:in `save!'
    	36: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `save!'
    	35: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
    	34: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:212:in `transaction'
    	33: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
    	32: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
    	31: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    	30: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    	29: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    	28: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    	27: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
    	26: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
    	25: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
    	24: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
    	23: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `block in save!'
    	22: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/validations.rb:53:in `save!'
    	21: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/persistence.rb:503:in `save!'
    	20: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/timestamp.rb:128:in `create_or_update'
    	19: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/callbacks.rb:327:in `create_or_update'
    	18: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
    	17: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:134:in `run_callbacks'
    	16: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `invoke_before'
    	15: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `each'
    	14: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `block in invoke_before'
    	13: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:201:in `block in halting'
    	12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:604:in `block in default_terminator'
    	11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:604:in `catch'
    	10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:605:in `block (2 levels) in default_terminator'
    	 9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting'
    	 8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'
    	 7: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
    	 6: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
    	 5: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
    	 4: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
    	 3: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
    	 2: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
    	 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final': OpenSSL::Cipher::CipherError

    How can I migrate my repository and must this be fixed ? @cody

  • @cody, this is what I have after restoring from another backup and running the migration script after upgrading to 13.9.5

    Traceback (most recent call last):
    	21: from bin/rails:4:in `<main>'
    	20: from bin/rails:4:in `require'
    	19: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/commands.rb:18:in `<top (required)>'
    	18: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/command.rb:46:in `invoke'
    	17: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/command/base.rb:69:in `perform'
    	16: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
    	15: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
    	14: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
    	13: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/commands/runner/runner_command.rb:42:in `perform'
    	12: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/commands/runner/runner_command.rb:42:in `load'
    	11: from /tmp/fix-legacy-hashed-storage-migration.rb:2:in `<top (required)>'
    	10: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
    	 9: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
    	 8: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
    	 7: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
    	 6: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
    	 5: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
    	 4: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
    	 3: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
    	 2: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
    	 1: from /tmp/fix-legacy-hashed-storage-migration.rb:9:in `block in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/storage_settings.rb:64:in `legacy_disk_path': git disk access denied (Gitlab::GitalyClient::StorageSettings::DirectPathAccessError)
    

    What are the other way to migrate those repos?

  • After some debugging and running the rb script in console directly, I was able to log this error message:

    #<Gitlab::GitalyClient::StorageSettings:0x00007f6bfa803a78 @legacy_disk_path="/var/opt/gitlab/git-data/repositories", @hash={"path"=>Gitlab::GitalyClient::StorageSettings::Deprecated, "gitaly_address"=>"unix:/var/opt/gitlab/gitaly/gitaly.socket"}>
    Traceback (most recent call last):
            3: from (irb):2
            2: from (irb):14:in `block in irb_binding'
            1: from lib/gitlab/gitaly_client/storage_settings.rb:64:in `legacy_disk_path'
    Gitlab::GitalyClient::StorageSettings::DirectPathAccessError (git disk access denied)

    So I presume this could be fixed somehow quite easily...

  • @d-kopriwa it looks like that's a simple permissions problem on git disk access. Make sure that the Gitaly service user has full permissions to /var/opt/gitlab/git-data/repositories.

  • I'm running into the same error on my 13.10 instance when trying this snippet. Looks like there's a feature flag was enabled that denies access by default now.

    @d-kopriwa we should be able to work around this by wrapping the request in a Gitlab::GitalyClient::StorageSettings.allow_disk_access do block:

    # Find all projects that GitLab thinks is in legacy storage
    Project.without_storage_feature(:repository).find_each(batch_size: 10) do |p|
    
        # Generate the hashed path
        hash_path = Storage::Hashed.new(p).disk_path
        storage = Gitlab.config.repositories.storages[p.repository.shard]
    
        # Enable access
        Gitlab::GitalyClient::StorageSettings.allow_disk_access do
    
            # Put a wildcard on the end so we find .git, .wiki.git and .design.git
            full_hashed_path = File.expand_path(File.join(storage.legacy_disk_path, hash_path + '*'))
    
            # If any repos already exist in hashed storage, complete the migration
            if(!Dir.glob(full_hashed_path).empty?)
                puts "Fixing ID:#{p.id} #{p.full_name} - #{full_hashed_path}"
            
                # Set the repo to read only if it isn't already
                if(!p.repository_read_only)
                    p.set_repository_read_only!
                end
    
                p.storage_version = ::Project::HASHED_STORAGE_FEATURES[:repository]
    
                p.reload_repository!
                p.write_repository_config
                p.track_project_repository
    
                p.repository_read_only = false
                p.save!(validate: false)
            end
        end
    end
  • @anton I think you should have edit access to this snippet. Can you update it to include the allow_disk_access block as in https://gitlab.com/-/snippets/2039252#note_553815038?

  • Hi @cody and thanks for your reply and help,

    @d-kopriwa it looks like that's a simple permissions problem on git disk access. Make sure that the Gitaly service user has full permissions to /var/opt/gitlab/git-data/repositories.

    I already checked with another instance of gitlab and permission are set accordingly, I believe the issue was with the script and @anton just posted a new version of it.

    This is the whole line I have run which use the fixed script for allow_disk_access error:

    wget -O /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2106558/raw
    gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb; gitlab-rake gitlab:storage:migrate_to_hashed; gitlab-rake gitlab:storage:list_legacy_projects; gitlab-rake gitlab:storage:list_legacy_attachments

    It doesn't crash anymore with the fix, however, I still have the following output not migrated:

    root@module:/# wget -O  /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2106558/raw && \
      gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb && \
      gitlab-rake gitlab:storage:migrate_to_hashed && \
      gitlab-rake gitlab:storage:list_legacy_projects && \
      gitlab-rake gitlab:storage:list_legacy_attachments
    
    Fixing ID:112 bootstrap-styled / bootstrap-styled-saga - /var/opt/gitlab/git-data/repositories/@hashed/b1/55/b1556dea32e9d0cdbfed038fd7787275775ea40939c146a64e205bcb349ad02f*
    Fixing ID:114 dev-tools / rollup-umd-scripts - /var/opt/gitlab/git-data/repositories/@hashed/9f/1f/9f1f9dce319c4700ef28ec8c53bd3cc8e6abe64c68385479ab89215806a5bdd6*
    Fixing ID:116 dev-tools / rollup-documentation - /var/opt/gitlab/git-data/repositories/@hashed/e5/b8/e5b861a6d8a966dfca7e7341cd3eb6be9901688d547a72ebed0b1f5e14f3d08d*
    Fixing ID:122 bootstrap-styled / bootstrap-styled-theme - /var/opt/gitlab/git-data/repositories/@hashed/1b/e0/1be00341082e25c4e251ca6713e767f7131a2823b0052caf9c9b006ec512f6cb*
    Fixing ID:129 dimitri.kopriwa / gitlab-ci-cli - /var/opt/gitlab/git-data/repositories/@hashed/65/66/6566230e3a3ce3774c1bbc7c18b590ae0f457bbcd511e90e3e7dca2a02e7addc*
    Fixing ID:165 rollup-umd / rollup-umd-ci-docker - /var/opt/gitlab/git-data/repositories/@hashed/bc/52/bc52dd634277c4a34a2d6210994a9a5e2ab6d33bb4a3a8963410e00ca6c15a02*
    Fixing ID:177 kopaxgroup / workstation - /var/opt/gitlab/git-data/repositories/@hashed/8c/d2/8cd2510271575d8430c05368315a87b9c4784c7389a47496080c1e615a2a00b6*
    Fixing ID:179 bootstrap-styled / v4 - /var/opt/gitlab/git-data/repositories/@hashed/30/68/3068430da9e4b7a674184035643d9e19af3dc7483e31cc03b35f75268401df77*
    Fixing ID:181 bootstrap-styled / utils - /var/opt/gitlab/git-data/repositories/@hashed/58/08/580811fa95269f3ecd4f22d176e079d36093573680b6ef66fa341e687a15b5da*
    Fixing ID:182 bootstrap-styled / redux - /var/opt/gitlab/git-data/repositories/@hashed/bf/a7/bfa7634640c53da7cb5e9c39031128c4e583399f936896f27f999f1d58d7b37e*
    Fixing ID:183 deploy-tools / ssh-deploy - /var/opt/gitlab/git-data/repositories/@hashed/b8/ae/b8aed072d29403ece56ae9641638ddd50d420f950bde0eefc092ee8879554141*
    Fixing ID:185 bootstrap-styled / provider - /var/opt/gitlab/git-data/repositories/@hashed/61/a2/61a229bae1e90331edd986b6bbbe617f7035de88a5bf7c018c3add6c762a6e8d*
    Fixing ID:186 bootstrap-styled / css-mixins - /var/opt/gitlab/git-data/repositories/@hashed/28/11/2811745d7b8d8874f6e653d176cefdd19e05e920ce389b9b7e83e5b2dfa546c7*
    Fixing ID:187 bootstrap-styled / toggle - /var/opt/gitlab/git-data/repositories/@hashed/38/b2/38b2d03f3256502b1e9db02b2d12aa27a46033ffe6d8c0ef0f2cf6b1530be9d8*
    Fixing ID:188 bootstrap-styled / css-utils - /var/opt/gitlab/git-data/repositories/@hashed/d6/06/d6061bbee6cf13bd73765faaea7cdd0af1323e4b125342ac346047f7c4bda1fc*
    Fixing ID:190 dimitri.kopriwa / samba - /var/opt/gitlab/git-data/repositories/@hashed/23/97/2397346b45823e070f6fc72ac94c0a999d234c472479f0e26b30cdf5942db854*
    Fixing ID:192 yeutech / www-assets - /var/opt/gitlab/git-data/repositories/@hashed/eb/3b/eb3be230bbd2844b1f5d8f2e4fab9ffba8ab22cfeeb69c4c1361993ba4f377b9*
    Fixing ID:194 rollup-umd / documentation-cli - /var/opt/gitlab/git-data/repositories/@hashed/75/59/7559ca4a957c8c82ba04781cd66a68d6022229fca0e8e88d8e487c96ee4446d0*
    Fixing ID:195 bootstrap-styled / rsg-components - /var/opt/gitlab/git-data/repositories/@hashed/1d/fa/1dfacb2ea5a03e0a915999e03b5a56196f1b1664d2f768d1b7eff60ac059789d*
    Fixing ID:196 rollup-umd / documentation - /var/opt/gitlab/git-data/repositories/@hashed/b4/bb/b4bbe448fde336bb6a7d7d765f36d3327c772b845e7b54c8282aa08c9775ddd7*
    Fixing ID:197 bootstrap-styled / documentation - /var/opt/gitlab/git-data/repositories/@hashed/8b/cb/8bcbb4c131df56f7c79066016241cc4bdf4e58db55c4f674e88b22365bd2e2ad*
    Fixing ID:198 yeutech / documentation - /var/opt/gitlab/git-data/repositories/@hashed/a4/e0/a4e00d7e6aa82111575438c5e5d3e63269d4c475c718b2389f6d02932c47f8a6*
    Fixing ID:199 bootstrap-styled / color - /var/opt/gitlab/git-data/repositories/@hashed/5a/39/5a39cadd1b007093db50744797c7a04a34f73b35ed444704206705b02597d6fd*
    Fixing ID:201 yeutech / react-router-dom-utils - /var/opt/gitlab/git-data/repositories/@hashed/43/97/43974ed74066b207c30ffd0fed5146762e6c60745ac977004bc14507c7c42b50*
    Fixing ID:202 yeutech / ra-language-intl - /var/opt/gitlab/git-data/repositories/@hashed/c1/7e/c17edaae86e4016a583e098582f6dbf3eccade8ef83747df9ba617ded9d31309*
    Fixing ID:203 bootstrap-styled / ra-ui - /var/opt/gitlab/git-data/repositories/@hashed/46/21/4621c1d55fa4e86ce0dae4288302641baac86dd53f76227c892df9d300682d41*
    Fixing ID:204 yeutech / react-admin-intl - /var/opt/gitlab/git-data/repositories/@hashed/fc/56/fc56dbc6d4652b315b86b71c8d688c1ccdea9c5f1fd07763d2659fde2e2fc49a*
    Fixing ID:206 yeutech / react-intl-polyfill - /var/opt/gitlab/git-data/repositories/@hashed/5c/f4/5cf4e26bd3d87da5e03f80a43a64f1220a1f4ba9e1d6348caea83c06353c3f39*
    Fixing ID:207 dimitri.kopriwa / example-tue - /var/opt/gitlab/git-data/repositories/@hashed/96/80/968076be2e38cf897d4d6cea3faca9c037e1a4e3b4b7744fb2533e07751bd30a*
    Fixing ID:208 yeutech / bitcoin-currency-converter - /var/opt/gitlab/git-data/repositories/@hashed/8d/f6/8df66f64b57424391d363fd6b811fed3c430c77597da265025728bd637bad804*
    Fixing ID:209 yeutech / test-polyfill - /var/opt/gitlab/git-data/repositories/@hashed/83/f8/83f814f7a92e365cbd79f9addceed185761a8d38a06a2d4350bb1fe4b7632b34*
    Fixing ID:211 kopaxgroup-api / user-management - /var/opt/gitlab/git-data/repositories/@hashed/09/34/093434a3ee9e0a010bb2c2aae06c2614dd24894062a1caf26718a01e175569b8*
    Fixing ID:214 kopaxgroup / documentation - /var/opt/gitlab/git-data/repositories/@hashed/80/2b/802b906a18591ead8a6dd809b262ace4c65c16e89764c40ae326cfcff811e10c*
    Fixing ID:215 kopaxgroup-api / site-service - /var/opt/gitlab/git-data/repositories/@hashed/d8/65/d86580a57f7bf542e85202283cb845953c9d28f80a8e651db08b2fc0b2d6a731*
    Fixing ID:217 kopaxgroup-api / um-users - /var/opt/gitlab/git-data/repositories/@hashed/16/ba/16badfc6202cb3f8889e0f2779b19218af4cbb736e56acadce8148aba9a7a9f8*
    Fixing ID:218 bootstrap-styled / ra-ui-extends - /var/opt/gitlab/git-data/repositories/@hashed/59/66/5966abd0cbfc86f98a186531b2b4ee5f6e910120ce13222f98207203dfc9a9a2*
    Fixing ID:219 yeutech / webpack-utils - /var/opt/gitlab/git-data/repositories/@hashed/31/4f/314f04b30f62e0056bd059354a5536fb2e302107eed143b5fa2aa0bbba07f608*
    Fixing ID:220 yeutech / app-context - /var/opt/gitlab/git-data/repositories/@hashed/36/79/36790ecd55c2030dc553685bef719df653f413a20cdad1bfd1dc934c76686ddd*
    Enqueuing migration of 16 projects in batches of 200. Done!
    * Found 16 projects using Legacy Storage
      - rollup-umd/rollup-umd-documentation-cli (id: 148)
        rollup-umd/rollup-umd-documentation-cli
      - rollup-umd/rollup-umd-ci-test (id: 150)
        rollup-umd/rollup-umd-ci-test
      - rollup-umd/rollup-umd-ci-release (id: 151)
        rollup-umd/rollup-umd-ci-release
      - rollup-umd/rollup-umd-ci-deploy (id: 152)
        rollup-umd/rollup-umd-ci-deploy
      - rollup-umd/rollup-umd-ci-release-transitive (id: 153)
        rollup-umd/rollup-umd-ci-release-transitive
      - yeutech/image-common (id: 161)
        yeutech/image-common
      - kopaxgroup/email-cli (id: 162)
        kopaxgroup/email-cli
      - basic/email-templates (id: 167)
        basic/email-templates
      - dka/email-templates-yeutech-cli (id: 169)
        dka/email-templates-yeutech-cli
      - yeutech/email-templates-yeutech (id: 171)
        yeutech/email-templates-yeutech
      - dev-tools/react-html-email-cli (id: 173)
        dev-tools/react-html-email-cli
      - yeutech/email-campaign-yeutech-cli (id: 174)
        yeutech/email-campaign-yeutech-cli
      - basic/language-provider-factory (id: 176)
        basic/language-provider-factory
      - yeutech/wallpapers-yeutech (id: 178)
        yeutech/wallpapers-yeutech
      - bootstrap-styled/v4.yeutech.com (id: 180)
        bootstrap-styled/v4.yeutech.com
      - deploy-tools/git-url-tweak (id: 184)
        deploy-tools/git-url-tweak
    * Found 0 attachments using Legacy Storage
    

    Any clue how I can fix all of them? I'd like to be able to keep GitLab long term up to date.

    Thanks !

    Edited by Kopax Anderson
  • @wchandler Sorry I was on OOO. I have updated the script in the snippet with your changes.

    @d-kopriwa It looks the snippet fixed alot of your projects. I assume that the fixed projects are visible in GitLab now? For these projects that are still in legacy storage:

      - rollup-umd/rollup-umd-documentation-cli (id: 148)
      - rollup-umd/rollup-umd-ci-test (id: 150)
      - rollup-umd/rollup-umd-ci-release (id: 151)
      - rollup-umd/rollup-umd-ci-deploy (id: 152)
      - rollup-umd/rollup-umd-ci-release-transitive (id: 153)
      - yeutech/image-common (id: 161)
      - kopaxgroup/email-cli (id: 162)
      - basic/email-templates (id: 167)
      - dka/email-templates-yeutech-cli (id: 169)
      - yeutech/email-templates-yeutech (id: 171)
      - dev-tools/react-html-email-cli (id: 173)
      - yeutech/email-campaign-yeutech-cli (id: 174)
      - basic/language-provider-factory (id: 176)
      - yeutech/wallpapers-yeutech (id: 178)
      - bootstrap-styled/v4.yeutech.com (id: 180)
      - deploy-tools/git-url-tweak (id: 184)
    1. Can you view these projects/files in the GitLab UI? Or does it show an empty repository?
    2. The migration might be failing for another reason. Can you run sudo gitlab-rake gitlab:storage:migrate_to_hashed to try and migrate these projects again? Afterwards, please wait 10 minutes to give the async jobs time to perform the migration. Then:
      • Run gitlab-rake gitlab:storage:list_legacy_projects to check if this fixed the problem (if it did, the projects won't appear in the list)
      • If not, let's look at the logs grep "HashedStorage::ProjectMigrateWorker" /var/log/gitlab/sidekiq/current (please attach any logs as files rather than copy/pasting the contents into your message). Chances are there is an error/stacktrace in here.
  • I appreciate your work creating this script! It fixed my inability to migrate the repos to hashed storage, but caused another issue -- any commit made in the last ~10 months disappeared from the repository fixed. 10 months might correspond the last time I made a major update (12 to 13).

    Has anyone else run into this problem, or have any idea why it happened? Any help is appreciated!

    Additional info:

    • Gitlab 13.12.3 (Omnibus install)
    • Gitlab is running behind Apache reverse proxy, NGINX is disabled, otherwise configuration is pretty much just the defaults
  • Huge disclaimer: Do not use this script unless you are 100% sure, and have a backup of your data ready to go! Unless you're in the exact same situation as I was, It has a high probability of destroying data.

    In continuation of my last post, in my case, it seems that ~10 months ago, Gitlab moved the repositories in the file system from legacy to hashed, but didn't identify that it had done so, which caused subsequent commits to be store in the legacy path. This all happened unbeknownst to me. Then, once I finally ran into an issue (trying to upgrade to Gitlab 14) I found this script. Running the script fixed the issue where Gitlab didn't identify that the repository was using hashed storage, but it didn't fix the subsequent commits stored in the legacy path because they shouldn't have been there anymore!

    I made a change to the original script that moves all of the files from the legacy path to the hashed path. In my case, it works because the legacy path contained the entire history of the repo. I'm not very familiar with Ruby, so it uses a system call instead of doing it the Ruby way, but hopefully it will at least help someone identify their problem.

    # Find all projects that GitLab thinks is in legacy storage
    Project.without_storage_feature(:repository).find_each(batch_size: 10) do |p|
    
        # Generate the hashed path
        hash_path = Storage::Hashed.new(p).disk_path
        storage = Gitlab.config.repositories.storages[p.repository.shard]
    
        # Enable access
        Gitlab::GitalyClient::StorageSettings.allow_disk_access do
    
            # Put a wildcard on the end so we find .git, .wiki.git and .design.git
            full_hashed_path = File.expand_path(File.join(storage.legacy_disk_path, hash_path + '*'))
    
            # If any repos already exist in hashed storage, complete the migration
            if(!Dir.glob(full_hashed_path).empty?)
                puts "Fixing ID:#{p.id} #{p.full_name} - #{full_hashed_path}"
    
                # Set the repo to read only if it isn't already
                if(!p.repository_read_only)
                    p.set_repository_read_only!
                end
    
                # Copy over git files from legacy path to hashed path
                if(!Dir.glob(File.expand_path(File.join(storage.legacy_disk_path, p.disk_path + '.git'))).empty?)
                    system("cp -rf #{File.expand_path(File.join(storage.legacy_disk_path, p.disk_path + '.git', '*'))} #{File.expand_path(File.join(storage.legacy_disk_path, hash_path + '.git'))}")
                end
    
                p.storage_version = ::Project::HASHED_STORAGE_FEATURES[:repository]
    
                p.reload_repository!
                p.write_repository_config
                p.track_project_repository
    
                p.repository_read_only = false
                p.save!(validate: false)
            end
        end
    end
  • @PBillodeau Thanks! I had the same issue: the script on the top of the page made some commits disappear. It looks like your version of the script fixed the issue. I restored the last backup, then ran your script, and I was able to migrate to GitLab 14.

    Edited by Alexis Lefebvre
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment