Migrate GitLab Pages to zip storage in the background
What does this MR do?
Currently, the only way to migrate pages to zip storage is to manually run the rake task:
gitlab-rake gitlab:pages:migrate_legacy_storage
It requires human intervention on migration and can take a long time to complete. (it took about a week on gitlab.com).
We can't remove this rake task because it allows users to control the process and fix some errors:
- it prints all errors by default, such as invalid symlinks and projects marked as deployed by absent on pages storage
- it also allows for fix these errors by ignoring invalid symlinks and marking projects as not deployed
- and it allows users to see the progress/check if everything has been migrated
However, making every instance admin run this task when upgrading to 14.0
may create too much work for them.
So we want to run the same migration code as the background migration.
This way we will migrate the data for most of the users without their intervention.
It still will be recommended to run the rake task and make sure everything has been migrated properly. But in most cases everything should be migrated automatically and even if users forgot to run the rake task everything should be fine.
We aim to merge this prior to 14.0, so we have more time to test it, and also for some users all the data will be migrated before they start upgrading to 14.0.
Currently, the background migration code uses services code, and these services rely on some models code. This goes against our guidelines https://docs.gitlab.com/ee/development/background_migrations.html#isolation.
But isolating this migration will require copying significant part of the code:
- models (I attempted this in the original version)
+ module Routable
+ def build_full_path
+ if parent && path
+ parent.build_full_path + '/' + path
+ else
+ path
+ end
+ end
+ end
+
+ # Namespace
+ class Namespace < ActiveRecord::Base
+ self.table_name = 'namespaces'
+ self.inheritance_column = :_type_disabled
+
+ include Routable
+
+ belongs_to :parent, class_name: "Namespace"
+ end
+
+ # copy class with only required code
+ class Project < ActiveRecord::Base
+ self.table_name = 'projects'
+ self.inheritance_column = :_type_disabled
+
+ include Routable
+
+ belongs_to :namespace
+ alias_method :parent, :namespace
+ alias_attribute :parent_id, :namespace_id
+
+ has_one :pages_metadatum, class_name: 'ProjectPagesMetadatum', inverse_of: :project
+
+ def pages_path
+ File.join(Settings.pages.path, build_full_path)
+ end
+ end
+
+ # copy class with only required code
+ class ProjectPagesMetadatum < ActiveRecord::Base
+ self.table_name = 'project_pages_metadata'
+ self.inheritance_column = :_type_disabled
+
+ include EachBatch
+
+ belongs_to :project, inverse_of: :pages_metadatum, class_name: 'Project'
+ belongs_to :pages_deployment, class_name: 'PagesDeployment'
+
+ scope :deployed, -> { where(deployed: true) }
+ scope :only_on_legacy_storage, -> { deployed.where(pages_deployment: nil) }
+ scope :with_project_route_and_deployment, -> { preload(:pages_deployment, project: [:namespace, :route]) }
+ end
+
+ # copy class with only required code
+ class PagesDeployment < ActiveRecord::Base
+ self.table_name = 'pages_deployments'
+ self.inheritance_column = :_type_disabled
+
+ include FileStoreMounter
+
+ attribute :file_store, :integer, default: -> { ::Pages::DeploymentUploader.default_store }
+
+ belongs_to :project, class_name: 'Project'
+
+ validates :file, presence: true
+ validates :file_store, presence: true, inclusion: { in: ObjectStorage::SUPPORTED_STORES }
+ validates :size, presence: true, numericality: { greater_than: 0, only_integer: true }
+ validates :file_count, presence: true, numericality: { greater_than_or_equal_to: 0, only_integer: true }
+ validates :file_sha256, presence: true
+
+ before_validation :set_size, if: :file_changed?
+
+ # Shall we copy all the uploader code migration as well?
+ mount_file_store_uploader ::Pages::DeploymentUploader
+
+ private
+
+ def set_size
+ self.size = file.size
+ end
+ end
- Also note, that
PagesDeployments
mounts::Pages::DeploymentUploader
, which in turn is based onGitlabUploader
,ObjectStorage::Concern
and requiresGitlab.config.pages
. - And we use services currently located under
app/services/pages/
I can try to move most of this under the lib/gitlab/background_migration/migrate_pages_to_zip_storage/
dir and reuse this code in the rake task, but I don't think it will make the migration more reliable.
Also the current approach was suggested in !54578 (comment 513143758)
Migration output
./bin/rails db:migrate:redo VERSION=20210302150310
== 20210302150310 ScheduleMigratePagesToZipStorage: reverting =================
== 20210302150310 ScheduleMigratePagesToZipStorage: reverted (0.0000s) ========
== 20210302150310 ScheduleMigratePagesToZipStorage: migrating =================
-- Scheduled 1 MigratePagesToZipStorage jobs with a maximum of 100 records per batch and an interval of 120 seconds.
The migration is expected to take at least 120 seconds. Expect all jobs to have completed after 2021-03-29 13:50:04 UTC."
== 20210302150310 ScheduleMigratePagesToZipStorage: migrated (0.0521s) ========
Database queries for the schedulling migration
SELECT "project_pages_metadata"."project_id"
FROM "project_pages_metadata"
WHERE "project_pages_metadata"."deployed" = TRUE
AND "project_pages_metadata"."pages_deployment_id" IS NULL
ORDER BY "project_pages_metadata"."project_id" ASC
LIMIT 1
https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/3119/commands/10272
SELECT "project_pages_metadata"."project_id"
FROM "project_pages_metadata"
WHERE "project_pages_metadata"."deployed" = TRUE
AND "project_pages_metadata"."pages_deployment_id" IS NULL
AND "project_pages_metadata"."project_id" >= 3707468
ORDER BY "project_pages_metadata"."project_id" ASC
LIMIT 1
OFFSET 100;
https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/3119/commands/10273
SELECT MIN(project_id),
MAX(project_id)
FROM "project_pages_metadata"
WHERE "project_pages_metadata"."deployed" = TRUE
AND "project_pages_metadata"."pages_deployment_id" IS NULL
AND "project_pages_metadata"."project_id" >= 3707468;
https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/3119/commands/10274
Database query for selecting batch(used in background migration)
SELECT "project_pages_metadata".*
FROM "project_pages_metadata"
WHERE "project_pages_metadata"."deployed" = TRUE
AND "project_pages_metadata"."pages_deployment_id" IS NULL
AND "project_pages_metadata"."project_id" BETWEEN 3707468 AND 24766231;
https://postgres.ai/console/gitlab/gitlab-production-tunnel/sessions/3518/commands/11644
Time estimation
We already migrated almost all of the gitlab.com(there are a few projects which failed to be deleted and marked as deployed), so in our own production this migration will be finished with only one iteration required, so it will take only a few minutes.
It took us about a week to run the migration by hand using 3 threads. So I'd say for instances with similar size this migration can take up to a months. (I don't think such instances exist though)
Rake migration updated output
vlad @ gdk1 ➜ gitlab git:(322001-poc-for-migrating-pages-to-zip-storage-in-the-background) ./bin/rake gitlab:pages:migrate_legacy_storage
I, [2021-04-15T14:45:26.167797 #1818942] INFO -- : Starting to migrate legacy pages storage to zip deployments
I, [2021-04-15T14:45:26.222246 #1818942] INFO -- : {:message=>"Pages legacy storage migration: Waiting for threads to finish..."}
E, [2021-04-15T14:45:26.859070 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gnuwget/wget2", :project_id=>3, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gnuwget/wget2", :duration=>0.11}
E, [2021-04-15T14:45:26.862268 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/demo-prepare", :project_id=>21, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/demo-prepare", :duration=>0.06}
E, [2021-04-15T14:45:26.862436 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/norbert/underscore", :project_id=>11, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/norbert/underscore", :duration=>0.12}
E, [2021-04-15T14:45:26.869745 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/Commit451/lab-coat", :project_id=>4, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/Commit451/lab-coat", :duration=>0.01}
E, [2021-04-15T14:45:26.872095 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test", :project_id=>23, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test", :duration=>0.01}
E, [2021-04-15T14:45:26.875534 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/marquita_kshlerin/wget2", :project_id=>12, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/marquita_kshlerin/wget2", :duration=>0.01}
E, [2021-04-15T14:45:26.882527 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/jashkenas/underscore", :project_id=>5, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/jashkenas/underscore", :duration=>0.01}
E, [2021-04-15T14:45:26.883321 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/neda.tromp/lab-coat", :project_id=>13, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/neda.tromp/lab-coat", :duration=>0.01}
E, [2021-04-15T14:45:26.886693 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/getsby-test-2", :project_id=>24, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/getsby-test-2", :duration=>0.01}
E, [2021-04-15T14:45:26.892480 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/flightjs/flight", :project_id=>6, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/flightjs/flight", :duration=>0.01}
E, [2021-04-15T14:45:26.895540 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/breana.schuster/lab-coat", :project_id=>14, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/breana.schuster/lab-coat", :duration=>0.01}
E, [2021-04-15T14:45:26.899355 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-3", :project_id=>25, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-3", :duration=>0.01}
E, [2021-04-15T14:45:26.905503 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/twitter/typeahead-js", :project_id=>7, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/twitter/typeahead-js", :duration=>0.01}
E, [2021-04-15T14:45:26.905667 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-4", :project_id=>26, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-4", :duration=>0.0}
E, [2021-04-15T14:45:26.908501 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/ronda/lab-coat", :project_id=>15, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/ronda/lab-coat", :duration=>0.01}
E, [2021-04-15T14:45:26.914845 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-5", :project_id=>27, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-5", :duration=>0.0}
E, [2021-04-15T14:45:26.918058 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/h5bp/html5-boilerplate", :project_id=>8, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/h5bp/html5-boilerplate", :duration=>0.01}
E, [2021-04-15T14:45:26.923015 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gerald_lindgren/underscore", :project_id=>16, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gerald_lindgren/underscore", :duration=>0.01}
E, [2021-04-15T14:45:26.924688 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/tony/lab-coat", :project_id=>9, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/tony/lab-coat", :duration=>0.0}
E, [2021-04-15T14:45:26.928938 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/objectstorage.pages.test", :project_id=>28, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/objectstorage.pages.test", :duration=>0.01}
E, [2021-04-15T14:45:26.935466 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/jama_grimes/underscore", :project_id=>17, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/jama_grimes/underscore", :duration=>0.01}
E, [2021-04-15T14:45:26.941866 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/alecia/wget2", :project_id=>10, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/alecia/wget2", :duration=>0.01}
E, [2021-04-15T14:45:26.942736 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/test-deployment-webhooks", :project_id=>30, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/test-deployment-webhooks", :duration=>0.01}
E, [2021-04-15T14:45:26.946356 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/serita.crona/lab-coat", :project_id=>18, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/serita.crona/lab-coat", :duration=>0.01}
E, [2021-04-15T14:45:26.953684 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-shell", :project_id=>2, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-shell", :duration=>0.01}
E, [2021-04-15T14:45:26.954405 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/pages-test", :project_id=>22, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/pages-test", :duration=>0.01}
E, [2021-04-15T14:45:26.955766 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/prepare-environments", :project_id=>19, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/prepare-environments", :duration=>0.01}
E, [2021-04-15T14:45:26.977514 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/prepare-demo", :project_id=>20, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/prepare-demo", :duration=>0.02}
I, [2021-04-15T14:45:26.977960 #1818942] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>0, :errored=>28}
I, [2021-04-15T14:45:27.112577 #1818942] INFO -- : {:message=>"Pages legacy storage migration: project migrated: ", :project_id=>1, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test", :duration=>0.16}
I, [2021-04-15T14:45:27.112742 #1818942] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>1, :errored=>28}
I, [2021-04-15T14:45:27.116354 #1818942] INFO -- : {:message=>"Pages legacy storage migration: project migrated: ", :project_id=>29, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/os-test", :duration=>0.16}
I, [2021-04-15T14:45:27.116664 #1818942] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>2, :errored=>28}
E, [2021-04-15T14:45:27.146838 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/webhooks-group/test-group-webhooks", :project_id=>31, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/webhooks-group/test-group-webhooks", :duration=>0.0}
E, [2021-04-15T14:45:27.149848 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/pages", :project_id=>34, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/pages", :duration=>0.0}
E, [2021-04-15T14:45:27.152698 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/new-pages", :project_id=>35, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/new-pages", :duration=>0.0}
E, [2021-04-15T14:45:27.155623 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-org.10.126.0.2.nip.io", :project_id=>33, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-org.10.126.0.2.nip.io", :duration=>0.0}
E, [2021-04-15T14:45:27.158060 #1818942] ERROR -- : {:message=>"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/pages-test", :project_id=>32, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/pages-test", :duration=>0.0}
I, [2021-04-15T14:45:27.197230 #1818942] INFO -- : {:message=>"Pages legacy storage migration: project migrated: ", :project_id=>36, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/new-new-pages", :duration=>0.04}
I, [2021-04-15T14:45:27.197382 #1818942] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>3, :errored=>33}
I, [2021-04-15T14:45:27.197970 #1818942] INFO -- : A total of 36 projects were processed.
I, [2021-04-15T14:45:27.198056 #1818942] INFO -- : - The 3 projects migrated successfully
I, [2021-04-15T14:45:27.198111 #1818942] INFO -- : - The 33 projects failed to be migrated
vlad @ gdk1 ➜ gitlab git:(322001-poc-for-migrating-pages-to-zip-storage-in-the-background) ✗ PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=true PAGES_MIGRATION_MARK_PROJECTS_AS_NOT_DEPLOYED=true ./bin/rake gitlab:pages:migrate_legacy_storage
I, [2021-04-15T14:46:06.655333 #1819371] INFO -- : Starting to migrate legacy pages storage to zip deployments
I, [2021-04-15T14:46:06.713816 #1819371] INFO -- : {:message=>"Pages legacy storage migration: Waiting for threads to finish..."}
I, [2021-04-15T14:46:07.307180 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test? Marked project as not deployed", :project_id=>23, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test", :duration=>0.07}
I, [2021-04-15T14:46:07.308834 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/norbert/underscore? Marked project as not deployed", :project_id=>11, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/norbert/underscore", :duration=>0.09}
I, [2021-04-15T14:46:07.314805 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/marquita_kshlerin/wget2? Marked project as not deployed", :project_id=>12, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/marquita_kshlerin/wget2", :duration=>0.08}
I, [2021-04-15T14:46:07.324846 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/neda.tromp/lab-coat? Marked project as not deployed", :project_id=>13, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/neda.tromp/lab-coat", :duration=>0.01}
I, [2021-04-15T14:46:07.325667 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/getsby-test-2? Marked project as not deployed", :project_id=>24, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/getsby-test-2", :duration=>0.01}
I, [2021-04-15T14:46:07.327178 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gnuwget/wget2? Marked project as not deployed", :project_id=>3, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gnuwget/wget2", :duration=>0.02}
I, [2021-04-15T14:46:07.337607 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/breana.schuster/lab-coat? Marked project as not deployed", :project_id=>14, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/breana.schuster/lab-coat", :duration=>0.01}
I, [2021-04-15T14:46:07.342196 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-3? Marked project as not deployed", :project_id=>25, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-3", :duration=>0.01}
I, [2021-04-15T14:46:07.345880 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/Commit451/lab-coat? Marked project as not deployed", :project_id=>4, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/Commit451/lab-coat", :duration=>0.02}
I, [2021-04-15T14:46:07.353219 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/ronda/lab-coat? Marked project as not deployed", :project_id=>15, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/ronda/lab-coat", :duration=>0.01}
I, [2021-04-15T14:46:07.365904 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-4? Marked project as not deployed", :project_id=>26, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-4", :duration=>0.02}
I, [2021-04-15T14:46:07.371397 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/jashkenas/underscore? Marked project as not deployed", :project_id=>5, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/jashkenas/underscore", :duration=>0.02}
I, [2021-04-15T14:46:07.372261 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gerald_lindgren/underscore? Marked project as not deployed", :project_id=>16, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gerald_lindgren/underscore", :duration=>0.02}
I, [2021-04-15T14:46:07.390802 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-5? Marked project as not deployed", :project_id=>27, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-5", :duration=>0.02}
I, [2021-04-15T14:46:07.392840 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/jama_grimes/underscore? Marked project as not deployed", :project_id=>17, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/jama_grimes/underscore", :duration=>0.01}
I, [2021-04-15T14:46:07.399543 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/flightjs/flight? Marked project as not deployed", :project_id=>6, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/flightjs/flight", :duration=>0.02}
I, [2021-04-15T14:46:07.409148 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/objectstorage.pages.test? Marked project as not deployed", :project_id=>28, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/objectstorage.pages.test", :duration=>0.02}
I, [2021-04-15T14:46:07.410922 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/serita.crona/lab-coat? Marked project as not deployed", :project_id=>18, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/serita.crona/lab-coat", :duration=>0.02}
I, [2021-04-15T14:46:07.421209 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/twitter/typeahead-js? Marked project as not deployed", :project_id=>7, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/twitter/typeahead-js", :duration=>0.02}
I, [2021-04-15T14:46:07.429162 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/demo-prepare? Marked project as not deployed", :project_id=>21, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/demo-prepare", :duration=>0.02}
I, [2021-04-15T14:46:07.432642 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/test-deployment-webhooks? Marked project as not deployed", :project_id=>30, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/test-deployment-webhooks", :duration=>0.02}
I, [2021-04-15T14:46:07.448571 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/h5bp/html5-boilerplate? Marked project as not deployed", :project_id=>8, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/h5bp/html5-boilerplate", :duration=>0.02}
I, [2021-04-15T14:46:07.455081 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/prepare-environments? Marked project as not deployed", :project_id=>19, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/prepare-environments", :duration=>0.02}
I, [2021-04-15T14:46:07.460717 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/pages-test? Marked project as not deployed", :project_id=>22, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/pages-test", :duration=>0.02}
I, [2021-04-15T14:46:07.475750 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/tony/lab-coat? Marked project as not deployed", :project_id=>9, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/tony/lab-coat", :duration=>0.02}
I, [2021-04-15T14:46:07.482210 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/prepare-demo? Marked project as not deployed", :project_id=>20, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/prepare-demo", :duration=>0.02}
I, [2021-04-15T14:46:07.484969 #1819371] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>26, :errored=>0}
I, [2021-04-15T14:46:07.485870 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/webhooks-group/test-group-webhooks? Marked project as not deployed", :project_id=>31, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/webhooks-group/test-group-webhooks", :duration=>0.02}
I, [2021-04-15T14:46:07.496882 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/alecia/wget2? Marked project as not deployed", :project_id=>10, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/alecia/wget2", :duration=>0.02}
I, [2021-04-15T14:46:07.513568 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/pages-test? Marked project as not deployed", :project_id=>32, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/pages-test", :duration=>0.01}
I, [2021-04-15T14:46:07.513888 #1819371] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>29, :errored=>0}
I, [2021-04-15T14:46:07.523019 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-shell? Marked project as not deployed", :project_id=>2, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-shell", :duration=>0.02}
I, [2021-04-15T14:46:07.523416 #1819371] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>30, :errored=>0}
I, [2021-04-15T14:46:07.579295 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/pages? Marked project as not deployed", :project_id=>34, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/pages", :duration=>0.01}
I, [2021-04-15T14:46:07.586227 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/new-pages? Marked project as not deployed", :project_id=>35, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/root/new-pages", :duration=>0.01}
I, [2021-04-15T14:46:07.594834 #1819371] INFO -- : {:message=>"Pages legacy storage migration: project migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-org.10.126.0.2.nip.io? Marked project as not deployed", :project_id=>33, :pages_path=>"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-org.10.126.0.2.nip.io", :duration=>0.01}
I, [2021-04-15T14:46:07.595163 #1819371] INFO -- : {:message=>"Pages legacy storage migration: batch processed", :migrated=>33, :errored=>0}
I, [2021-04-15T14:46:07.595741 #1819371] INFO -- : A total of 33 projects were processed.
I, [2021-04-15T14:46:07.595974 #1819371] INFO -- : - The 33 projects migrated successfully
I, [2021-04-15T14:46:07.596187 #1819371] INFO -- : - The 0 projects failed to be migrated
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Background migration logs
(I changed batch size to 20, and time to 2 minutes to speed up tests locally)
vlad @ gdk1 ➜ gitlab git:(322001-poc-for-migrating-pages-to-zip-storage-in-the-background) ✗ tail -F log/application_json.log | grep "legacy"
{"severity":"ERROR","time":"2021-04-15T13:03:11.071Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/norbert/underscore","project_id":11,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/norbert/underscore","duration":0.02}
{"severity":"ERROR","time":"2021-04-15T13:03:11.074Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/marquita_kshlerin/wget2","project_id":12,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/marquita_kshlerin/wget2","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.077Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gnuwget/wget2","project_id":3,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/gnuwget/wget2","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.080Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/neda.tromp/lab-coat","project_id":13,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/neda.tromp/lab-coat","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.085Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/breana.schuster/lab-coat","project_id":14,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/breana.schuster/lab-coat","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.090Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/Commit451/lab-coat","project_id":4,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/Commit451/lab-coat","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.094Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/ronda/lab-coat","project_id":15,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/ronda/lab-coat","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.097Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/jashkenas/underscore","project_id":5,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/jashkenas/underscore","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.100Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gerald_lindgren/underscore","project_id":16,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/gerald_lindgren/underscore","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.103Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/flightjs/flight","project_id":6,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/flightjs/flight","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.105Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/jama_grimes/underscore","project_id":17,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/jama_grimes/underscore","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.108Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/serita.crona/lab-coat","project_id":18,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/serita.crona/lab-coat","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.111Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/twitter/typeahead-js","project_id":7,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/twitter/typeahead-js","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.114Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/h5bp/html5-boilerplate","project_id":8,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/h5bp/html5-boilerplate","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.117Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/prepare-environments","project_id":19,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/prepare-environments","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.120Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/tony/lab-coat","project_id":9,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/tony/lab-coat","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.122Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/prepare-demo","project_id":20,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/prepare-demo","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.126Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/alecia/wget2","project_id":10,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/alecia/wget2","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:03:11.128Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-shell","project_id":2,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-shell","duration":0.0}
{"severity":"INFO","time":"2021-04-15T13:03:11.246Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: project migrated: ","project_id":1,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test","duration":0.12}
{"severity":"INFO","time":"2021-04-15T13:03:11.246Z","correlation_id":"1c65f633435e469053be3b1ab182eed7","message":"Pages legacy storage migration: batch processed","migrated":1,"errored":19}
{"severity":"ERROR","time":"2021-04-15T13:05:10.402Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test","project_id":23,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.408Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/getsby-test-2","project_id":24,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/getsby-test-2","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.413Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-3","project_id":25,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-3","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.418Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-4","project_id":26,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-4","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.421Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-5","project_id":27,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/gatsby-test-5","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.425Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/objectstorage.pages.test","project_id":28,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/objectstorage.pages.test","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.428Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/test-deployment-webhooks","project_id":30,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/test-deployment-webhooks","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.431Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/demo-prepare","project_id":21,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/demo-prepare","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.433Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/pages-test","project_id":22,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/pages-test","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.435Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/webhooks-group/test-group-webhooks","project_id":31,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/webhooks-group/test-group-webhooks","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.437Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/pages-test","project_id":32,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/pages-test","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.439Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/pages","project_id":34,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/pages","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.441Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/root/new-pages","project_id":35,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/new-pages","duration":0.0}
{"severity":"ERROR","time":"2021-04-15T13:05:10.443Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project failed to be migrated: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-org.10.126.0.2.nip.io","project_id":33,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-org.10.126.0.2.nip.io","duration":0.0}
{"severity":"INFO","time":"2021-04-15T13:05:10.489Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project migrated: ","project_id":29,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/os-test","duration":0.04}
{"severity":"INFO","time":"2021-04-15T13:05:10.529Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: project migrated: ","project_id":36,"pages_path":"/home/vlad/gdk/gitlab/shared/pages/root/new-new-pages","duration":0.04}
{"severity":"INFO","time":"2021-04-15T13:05:10.530Z","correlation_id":"d1a19333ea805745e3cd74f41af76f07","message":"Pages legacy storage migration: batch processed","migrated":2,"errored":14}
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Related to #322001 (closed)
Merge request reports
Activity
2 Warnings New migrations added but db/structure.sql wasn't updated Usually, when adding new migrations, db/structure.sql should be
updated too (unless the migration isn't changing the DB schema
and isn't the most recent one).Please add a merge request type to this merge request. Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.
To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
Category Reviewer Maintainer backend Michał Zając ( @Quintasan
) (UTC+2, 1 hour behind@vshushlin
)Imre Farkas ( @ifarkas
) (UTC+2, 1 hour behind@vshushlin
)database Andy Soiron ( @Andysoiron
) (UTC+2, 1 hour behind@vshushlin
)Steve Abrams ( @sabrams
) (UTC-6, 9 hours behind@vshushlin
)~migration No reviewer available No maintainer available If needed, you can retry the
danger-review
job that generated this comment.Generated by
DangerEdited by 🤖 GitLab Bot 🤖added sectionops label
added 168 commits
-
6e1ece7f...14c54040 - 167 commits from branch
master
- 86cf9d4f - WIP
-
6e1ece7f...14c54040 - 167 commits from branch
- Resolved by Vladimir Shushlin
@ayufan @alipniagov this is WIP and very far from being completed, but I wanted to see your thoughts on the general approach
assigned to @ayufan and @alipniagov and unassigned @vshushlin
- Resolved by Vladimir Shushlin
unassigned @ayufan
mentioned in issue gitlab-com/www-gitlab-com#7702 (closed)
assigned to @vshushlin and unassigned @alipniagov
added 1218 commits
-
cd29479a...31dddbc1 - 1217 commits from branch
master
- 85b11836 - WIP
-
cd29479a...31dddbc1 - 1217 commits from branch
added databasereview pending label
added 205 commits
-
ca94dcb1...7539bb17 - 204 commits from branch
master
- b89ab0a9 - Add pages zip storage background migration
-
ca94dcb1...7539bb17 - 204 commits from branch
added 2319 commits
-
e6af34c6...87f8d02d - 2317 commits from branch
master
- c63cc794 - Add pages zip storage background migration
- aa9a32dc - WIP
-
e6af34c6...87f8d02d - 2317 commits from branch
added 1 commit
- c8fadccd - Add migration scheduling pages zip storage migration
added 1 commit
- 5d0bc221 - Add migration scheduling pages zip storage migration
added 294 commits
-
bde1b3d2...9c471988 - 293 commits from branch
master
- 9dd54a20 - Automatically migrate pages to zip storage
-
bde1b3d2...9c471988 - 293 commits from branch
changed milestone to %13.11
- Resolved by Vladimir Shushlin
- Resolved by Kamil Trzciński
- Resolved by Vladimir Shushlin
@ebaque can you take an initial look at this MR?
assigned to @ebaque
- Resolved by Etienne Baqué
- Resolved by Vladimir Shushlin
- Resolved by Vladimir Shushlin
- Resolved by Vladimir Shushlin
- Resolved by Vladimir Shushlin
- Resolved by Vladimir Shushlin
unassigned @ebaque
requested review from @ebaque
added 1958 commits
-
ad957154...89e711cb - 1956 commits from branch
master
- 81e35146 - Automatically migrate pages to zip storage
- feacae02 - WIP
-
ad957154...89e711cb - 1956 commits from branch
- Resolved by Vladimir Shushlin