Avoid idling in transaction while saving project export object
When large exports were generated, we saw in production that these files
often were uploaded to object storage but were missing the relevant
import_export_uploads
database entry. This was happening because the
idle in transaction timeout killed the database transaction.
CarrierWave v1, which we run now, will idle in transaction while saving
a file to object storage. CarrierWave v1 saves the file in the
after_save
callback. From the database perspective, this begins a
transaction, inserts an entry into import_export_uploads
, and then
attempts to upload the file to object storage. However, this quickly
runs into our 60-second idle-in-transaction timeout.
CarrierWave v2 tried to fix this in
https://github.com/carrierwaveuploader/carrierwave/pull/2209 by moving
the object storage save in the after_commit
callback. This means the
object would only be uploaded after the database COMMIT, but this
presents other problems. For example, the file may appear to be available
even though it hasn't finished uploading to object storage.
CarrierWave v3, which has not yet released, will revert back to the v1 behavior via https://github.com/carrierwaveuploader/carrierwave/pull/2546 with a twist: if the transaction is rolled back for reason, CarrierWave will delete the file in object storage.
For project exports, the v2 issues aren't a big deal since we don't need
to read the value after save, but this does mean the actual file may not
appear until some time after the database entry has been committed. This
means that the UI might display that a project export is available even
though it hasn't finished uploading, resulting in a 404. To avoid that,
we might want to call file.exist?
, but this would require an HTTP HEAD
request.
Since we don't want to make this HEAD request whenever we show the project settings, we now will check for this condition whenever a user requests to download the export. If the database entry exists but the file does not exist, the user will see this:
Relates to gitlab-com/gl-infra/production#4814 (closed), #195981 (closed)