Skip to content

Add dependency proxy migrate rake task

Anton Smith requested to merge feature/dependency-proxy-migrate-rake into master

What does this MR do and why?

Describe in detail what your merge request does and why.

Adds a gitlab-rake "gitlab:dependency_proxy:migrate command that can be used to migrate dependency proxy blobs/manifests from local storage to object storage.

Closes #343064 (closed)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

❯ rake gitlab:dependency_proxy:migrate
WARNING: This version of GitLab depends on gitlab-shell 13.23.1, but you're running 13.23.0. Please update gitlab-shell.
I, [2022-02-02T06:43:02.227173 #46067]  INFO -- : Starting transfer of dependency proxy files to object storage
I, [2022-02-02T06:43:02.296297 #46067]  INFO -- : Transferred dependency proxy blob file 33 of size 2142 to object storage
I, [2022-02-02T06:43:03.334397 #46067]  INFO -- : Transferred dependency proxy blob file 34 of size 83518086 to object storage
I, [2022-02-02T06:43:03.341942 #46067]  INFO -- : Transferred dependency proxy blob file 35 of size 2754 to object storage
I, [2022-02-02T06:43:04.301943 #46067]  INFO -- : Transferred dependency proxy blob file 36 of size 76097157 to object storage
I, [2022-02-02T06:43:04.308582 #46067]  INFO -- : Transferred dependency proxy blob file 37 of size 1797 to object storage
I, [2022-02-02T06:43:04.316423 #46067]  INFO -- : Transferred dependency proxy blob file 38 of size 149 to object storage
I, [2022-02-02T06:43:04.323179 #46067]  INFO -- : Transferred dependency proxy blob file 39 of size 3055 to object storage
I, [2022-02-02T06:43:04.330732 #46067]  INFO -- : Transferred dependency proxy blob file 40 of size 129 to object storage
I, [2022-02-02T06:43:04.337331 #46067]  INFO -- : Transferred dependency proxy blob file 41 of size 199 to object storage
I, [2022-02-02T06:43:04.343726 #46067]  INFO -- : Transferred dependency proxy blob file 42 of size 4719 to object storage
I, [2022-02-02T06:43:04.352446 #46067]  INFO -- : Transferred dependency proxy blob file 43 of size 893 to object storage
I, [2022-02-02T06:43:04.359804 #46067]  INFO -- : Transferred dependency proxy blob file 44 of size 601 to object storage
I, [2022-02-02T06:43:04.523540 #46067]  INFO -- : Transferred dependency proxy blob file 45 of size 25352768 to object storage
I, [2022-02-02T06:43:04.530054 #46067]  INFO -- : Transferred dependency proxy blob file 46 of size 666 to object storage
I, [2022-02-02T06:43:04.536611 #46067]  INFO -- : Transferred dependency proxy blob file 47 of size 1394 to object storage
I, [2022-02-02T06:43:04.544766 #46067]  INFO -- : Transferred dependency proxy blob file 48 of size 602 to object storage
I, [2022-02-02T06:43:04.551395 #46067]  INFO -- : Transferred dependency proxy blob file 49 of size 895 to object storage
I, [2022-02-02T06:43:04.558428 #46067]  INFO -- : Transferred dependency proxy blob file 50 of size 667 to object storage
I, [2022-02-02T06:43:04.566299 #46067]  INFO -- : Transferred dependency proxy blob file 51 of size 1394 to object storage
I, [2022-02-02T06:43:04.591874 #46067]  INFO -- : Transferred dependency proxy blob file 52 of size 2818413 to object storage
I, [2022-02-02T06:43:04.600009 #46067]  INFO -- : Transferred dependency proxy blob file 53 of size 1471 to object storage
I, [2022-02-02T06:43:04.607452 #46067]  INFO -- : Transferred dependency proxy blob file 54 of size 1456 to object storage
I, [2022-02-02T06:43:04.632813 #46067]  INFO -- : Transferred dependency proxy manifest file 19 of size 529 to object storage
I, [2022-02-02T06:43:04.639150 #46067]  INFO -- : Transferred dependency proxy manifest file 20 of size 529 to object storage
I, [2022-02-02T06:43:04.646338 #46067]  INFO -- : Transferred dependency proxy manifest file 21 of size 529 to object storage
I, [2022-02-02T06:43:04.653067 #46067]  INFO -- : Transferred dependency proxy manifest file 22 of size 529 to object storage
I, [2022-02-02T06:43:04.661002 #46067]  INFO -- : Transferred dependency proxy manifest file 23 of size 3038 to object storage
I, [2022-02-02T06:43:04.668028 #46067]  INFO -- : Transferred dependency proxy manifest file 24 of size 1570 to object storage
I, [2022-02-02T06:43:04.676604 #46067]  INFO -- : Transferred dependency proxy manifest file 25 of size 1570 to object storage
I, [2022-02-02T06:43:04.683571 #46067]  INFO -- : Transferred dependency proxy manifest file 26 of size 528 to object storage
I, [2022-02-02T06:43:04.690473 #46067]  INFO -- : Transferred dependency proxy manifest file 27 of size 527 to object storage

How to set up and validate locally

  1. Setup GDK with the registry. Do not enable object storage at this point.

  2. In the GitLab UI, create a proxy group. Pull some images into the dependency proxy:

    ❯ docker pull gdk.test:3000/proxy/dependency_proxy/containers/alpine:latest
.......

  3. Confirm via bundle exec rails console that the blobs/manifests are stored locally:

    g = Group.find_by_full_path('proxy')
g.dependency_proxy_blobs.each do |b|
  puts "#{b.file_store}, #{b.file.path}"
end; nil
g.dependency_proxy_manifests.each do |m|
  puts "#{m.file_store}, #{m.file.path}"
end; nil
    # Example output for blob 34
1, /Users/antons/gitlab/gitlab-development-kit/gitlab/shared/dependency_proxy/0f/d4/0fd42b3f73c448b34940b339f87d07adf116b05c0227aad72e8f0ee90533e699/dependency_proxy/109/files/34/a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1.gz

# Example output for manifest 26
1, /Users/antons/gitlab/gitlab-development-kit/gitlab/shared/dependency_proxy/0f/d4/0fd42b3f73c448b34940b339f87d07adf116b05c0227aad72e8f0ee90533e699/dependency_proxy/109/files/26/alpine:latest.json

    For extra validation, the paths reported can be checked on disk with ls.

  4. Reconfigure GDK with object storage using the built in Minio, and restart GDK.

  5. Run the rake task to migrate dependency proxy objects:

    ❯ rake gitlab:dependency_proxy:migrate

  6. Confirm via bundle exec rails console that the blobs/manifests are stored in object storage:

    g = Group.find_by_full_path('proxy')
g.dependency_proxy_blobs.each do |b|
  puts "#{b.file_store}, #{b.file.path}"
end; nil
g.dependency_proxy_manifests.each do |m|
  puts "#{m.file_store}, #{m.file.path}"
end; nil
    # Example output for blob 34
2, 0f/d4/0fd42b3f73c448b34940b339f87d07adf116b05c0227aad72e8f0ee90533e699/dependency_proxy/109/files/34/a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1.gz

# Example output for manifest 26
2, 0f/d4/0fd42b3f73c448b34940b339f87d07adf116b05c0227aad72e8f0ee90533e699/dependency_proxy/109/files/26/alpine:latest.json

    For extra validation, the paths reported can be checked via the Minio client mc ls and also check if the former paths exist locally:

    ❯ mc ls gdk/dependency-proxy/0f/d4/0fd42b3f73c448b34940b339f87d07adf116b05c0227aad72e8f0ee90533e699/dependency_proxy/109/files/26/alpine:latest.json
[2022-02-02 06:43:04 NZDT]   528B STANDARD alpine:latest.json

  7. Additionally, let's ensure that we can still pull the same image from the dependency proxy:

    # Clear the local cache
docker rmi $(docker images -a -q)

# Pull the image
docker pull gdk.test:3000/proxy/dependency_proxy/containers/alpine:latest

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Anton Smith

Merge request reports