Cannot move issues containing uploads when using Ceph 12.2.13

Summary

Versions affected - 12.3.5-ee and 12.3.6-ee

When moving an issue containing uploads (when using Ceph 12.2.13), the move fails with a An error occurred while moving the issue error.

The logs will show a ActiveRecord::RecordInvalid","exception.message":"Validation failed: Size can't be blank error.

Full stacktrace:

      "app/uploaders/records_uploads.rb:33:in `tap'",
      "app/uploaders/records_uploads.rb:33:in `readd_upload'",
      "app/uploaders/records_uploads.rb:26:in `block in record_upload'",
      "app/uploaders/records_uploads.rb:26:in `record_upload'",
      "app/uploaders/file_uploader.rb:183:in `copy_file'",
      "app/uploaders/file_uploader.rb:171:in `copy_to'",
      "lib/gitlab/gfm/uploads_rewriter.rb:32:in `block in rewrite'",
      "lib/gitlab/gfm/uploads_rewriter.rb:24:in `gsub'",
      "lib/gitlab/gfm/uploads_rewriter.rb:24:in `rewrite'",
      "app/services/markdown_content_rewriter_service.rb:25:in `block in execute'",
      "app/services/markdown_content_rewriter_service.rb:23:in `each'",
      "app/services/markdown_content_rewriter_service.rb:23:in `inject'",
      "app/services/markdown_content_rewriter_service.rb:23:in `execute'",
      "app/services/issuable/clone/base_service.rb:48:in `update_new_entity_description'",
      "app/services/issuable/clone/base_service.rb:36:in `update_new_entity'",
      "app/services/issuable/clone/base_service.rb:19:in `block in execute'",
      "app/services/issuable/clone/base_service.rb:16:in `execute'",
      "app/services/issues/move_service.rb:18:in `execute'",
      "lib/gitlab/metrics/instrumentation.rb:161:in `block in execute'",
      "lib/gitlab/metrics/method_call.rb:27:in `measure'",
      "lib/gitlab/metrics/instrumentation.rb:161:in `execute'",
      "app/services/issues/update_service.rb:109:in `move_issue_to_new_project'",
      "lib/gitlab/metrics/instrumentation.rb:161:in `block in move_issue_to_new_project'",
      "lib/gitlab/metrics/method_call.rb:27:in `measure'",
      "lib/gitlab/metrics/instrumentation.rb:161:in `move_issue_to_new_project'",
      "app/services/issues/update_service.rb:11:in `execute'",
      "ee/app/services/ee/issues/update_service.rb:13:in `execute'",
      "app/controllers/projects/issues_controller.rb:146:in `move'",
      "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
      "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'",
      "app/controllers/application_controller.rb:491:in `set_current_admin'",
      "lib/gitlab/session.rb:11:in `with_session'",
      "app/controllers/application_controller.rb:482:in `set_session_storage'",
      "lib/gitlab/i18n.rb:73:in `with_locale'",
      "lib/gitlab/i18n.rb:79:in `with_user_locale'",
      "app/controllers/application_controller.rb:476:in `set_locale'",
      "lib/gitlab/error_tracking.rb:50:in `with_context'",
      "app/controllers/application_controller.rb:541:in `sentry_context'",
      "app/controllers/application_controller.rb:469:in `block in set_current_context'",
      "lib/gitlab/application_context.rb:52:in `block in use'",
      "lib/gitlab/application_context.rb:52:in `use'",
      "lib/gitlab/application_context.rb:20:in `with_context'",
      "app/controllers/application_controller.rb:462:in `set_current_context'",
      "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
      "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
      "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
      "lib/gitlab/metrics/transaction.rb:61:in `run'",
      "lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
      "lib/gitlab/request_profiler/middleware.rb:17:in `call'",
      "ee/lib/gitlab/jira/middleware.rb:19:in `call'",
      "lib/gitlab/middleware/go.rb:20:in `call'",
      "lib/gitlab/etag_caching/middleware.rb:13:in `call'",
      "lib/gitlab/middleware/multipart.rb:140:in `call'",
      "lib/gitlab/middleware/read_only/controller.rb:51:in `call'",
      "lib/gitlab/middleware/read_only.rb:18:in `call'",
      "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
      "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
      "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
      "lib/gitlab/middleware/request_context.rb:23:in `call'",
      "config/initializers/fix_local_cache_middleware.rb:9:in `call'",
      "lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'",
      "lib/gitlab/middleware/release_env.rb:12:in `call'"

Steps to reproduce

Configure GitLab to use Ceph 12.2.13 as the object storage for uploads:

gitlab_rails['uploads_object_store_enabled'] = true
gitlab_rails['uploads_object_store_direct_upload'] = false
gitlab_rails['uploads_object_store_proxy_download'] = true
gitlab_rails['uploads_object_store_remote_directory'] = 'XXX'
gitlab_rails['uploads_object_store_connection'] = {
    'provider' => 'AWS',
    'region' => 'XXX',
    'aws_access_key_id' => 'XXX',
    'aws_secret_access_key' => 'XXX',
    # The below options configure an S3 compatible host instead of AWS
    'host' => 'XXX:XXX',
    'endpoint' => 'XXX:XXX/swift/v1',
    'path_style' => true,
    'aws_signature_version' => 2
}

Create a new project, and then create a new issue. Ensure that a file is uploaded into the issue description.
Verify that the file has been uploaded to object storage. Take note of the size.
Attempt to move the issue.

Debugging

It appears moving an issue actually reuploads any uploads (attachments) on the issue. Using the following code, we can narrow down what to look for in sudo gitlab-rails c:

# Get your project
p = Project.find(7)

# Get the last file uploaded to the project (Untitled_Diagram.png)
u = p.uploads.last

# Get the path (and manually check it exists in Ceph)
u.path

# Attempt to upload the file from Ceph (the move operation)
uf = UploaderFinder.new(p, u.secret, "Untitled_Diagram.png").execute

# Check the file size (returned nil strangely)
uf.file.size

# Print the raw file
uf.file

Using the code above, we can see the file size embedded within uf.file, but uf.file.size returns nil.

At this point, we don't know if this is a bug in Ceph, GitLab or one of the underlying gems.

What is the current bug behavior?

It is not possible to move issues containing uploads on Ceph Radosgw S3.

What is the expected correct behavior?

It should be possible to move issues containing uploads on Ceph Radosgw S3.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Workaround

Remove uploads from the issue description before moving the issue.

Possible fixes

Edited Feb 22, 2021 by Anton Smith