Packages uploads fails with: object storage enabled, direct upload disabled, background upload enabled

backend-weight2

Problem to solve

This is a follow-up of !26387 (comment 298710168) (See case 6.)

When:

• Object Storage enabled
• Direct Upload disabled
• Background Upload enabled

in the packages section of the config we get:

Sidekiq::Worker::EnqueueFromTransactionError (`ObjectStorage::BackgroundMoveWorker.perform_async` cannot be called inside a transaction as this can lead to
race conditions when the worker runs before the transaction is committed and
tries to access a model that has not been saved yet.

Use an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.
):
  /Users/david/projects/gitlab-development-kit/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:23:in `block (2 levels) in <module:NoEnqueueingFromTransactions>'
  /Users/david/projects/gitlab-development-kit/gitlab/app/uploaders/object_storage.rb:270:in `schedule_background_upload'

Reach

• This is happening on NuGet and NPM packages all package types.

Suggested Fix

Use include ObjectStorage::BackgroundMove in Packages::PackageFile in place of having after :store, :schedule_background_upload in Packages::PackageFileUploader (according to https://docs.gitlab.com/ee/development/file_storage.html#using-objectstorageextensionrecordsuploads).

It fixes the error above but it messes with the NuGet background worker. Basically, the background move worker and the NuGet background worker try both to access the file and that's not possible due to the exclusive lease (https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/app/uploaders/object_storage.rb#L250).

We will need a way to properly queue both worker jobs in sequence and not in parallel.