Skip to content

Fix object deletion not working with Azure Blob Storage

Stan Hu requested to merge sh-fix-azure-delete into master

What does this MR do and why?

When deleting a group, we have a wrapper that attempts to use a Fog providers's #delete_object method. While fog-google and fog-aws supported this, gitlab-fog-azure-rm never did. This commit upgrades the gitlab-fog-azure-rm gem to v1.7.0 and adds basic acceptance tests for the various Fog providers. Note that only fog-aws actually mocks object deletion properly, so we have to be content with #delete_object returning a truthy value.

This merge request also fixes object storage bucket prefix handling for blob deletion. In !91307 (merged), we added formal support for using bucket prefixes in object storage. However, deletion of objects did not handle this. This commit appends the prefix to the object key if it is available.

Relates to #391461 (closed)

How to set up and validate locally

  1. Configure Azure object storage support. In gdk.yml, I have:
object_store:
  consolidated_form: true
  enabled: true
  connection:
    provider: AzureRM
    azure_storage_account_name: REDACTED_NAME
    azure_storage_access_key: REDACTED_KEY
  objects:
    artifacts:
      bucket: test1/artifacts
    external_diffs:
      bucket: test1/external_diffs
    lfs:
      bucket: test1/lfs
    uploads:
      bucket: test1/uploads
    packages:
      bucket: test1/packages
    dependency_proxy:
      bucket: test1/dependency-proxy
    terraform_state:
      bucket: test1/terraform
    pages:
      bucket: test1/pages
  1. Run gdk reconfigure.
  2. Restart GitLab: gdk restart (rails-background-jobs, rails, and gitlab-workhorse need to be restarted)
  3. In the GDK, create a group and a project.
  4. In the project, create an issue and attach an image.
  5. Delete the group.
  6. By default, the group won't be deleted for 7 days. Go back to the group Advanced settings, and confirm the deletion of the group.
  7. View the Sidekiq /admin/sidekiq/morgue page to verify no errors.

On master, the error shows up:

image

With this branch, no error shows up. I also confirmed in the Azure blob browser that the blob was deleted. Before:

image

After:

image

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Stan Hu

Merge request reports