Skip to content

Improve error handling and logging in pages migration

What does this MR do?

This rake task is used to migrate files stored on disk to object storage.

It just enumerates all the projects which are not yet migrated, zips content and uploads it to the PagesDeployment model.

Currently, all errors are reported only to Sentry: https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+%22app%2Fservices%2Fpages%22, and all invalid files/links on disk are simply ignored.

This MR makes these errors:

  1. be reported to the migration output
  2. prevent project from being migrated.

Our plan is:

  1. delete all already migrated data using clean_migrated_zip_storage introduced in !53727 (merged)
  2. rerun the migration with 'PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=false' (by default)
  3. inspect all the errors in logs
  4. rerun the migration with 'PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=false' when we'll be sure that all the errors are legit

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #321037 (closed)

Edited by Kamil Trzciński

Merge request reports