Skip to content

Avoid idling in transaction while saving project export object

Stan Hu requested to merge sh-project-exports-avoid-idling-in-transaction into master

When large exports were generated, we saw in production that these files often were uploaded to object storage but were missing the relevant import_export_uploads database entry. This was happening because the idle in transaction timeout killed the database transaction.

CarrierWave v1, which we run now, will idle in transaction while saving a file to object storage. CarrierWave v1 saves the file in the after_save callback. From the database perspective, this begins a transaction, inserts an entry into import_export_uploads, and then attempts to upload the file to object storage. However, this quickly runs into our 60-second idle-in-transaction timeout.

CarrierWave v2 tried to fix this in https://github.com/carrierwaveuploader/carrierwave/pull/2209 by moving the object storage save in the after_commit callback. This means the object would only be uploaded after the database COMMIT, but this presents other problems. For example, the file may appear to be available even though it hasn't finished uploading to object storage.

CarrierWave v3, which has not yet released, will revert back to the v1 behavior via https://github.com/carrierwaveuploader/carrierwave/pull/2546 with a twist: if the transaction is rolled back for reason, CarrierWave will delete the file in object storage.

For project exports, the v2 issues aren't a big deal since we don't need to read the value after save, but this does mean the actual file may not appear until some time after the database entry has been committed. This means that the UI might display that a project export is available even though it hasn't finished uploading, resulting in a 404. To avoid that, we might want to call file.exist?, but this would require an HTTP HEAD request.

Since we don't want to make this HEAD request whenever we show the project settings, we now will check for this condition whenever a user requests to download the export. If the database entry exists but the file does not exist, the user will see this:

image

Relates to gitlab-com/gl-infra/production#4814 (closed), #195981 (closed)

Edited by Stan Hu

Merge request reports