Merge request diffs return 500 error
Summary
I started Gitlab with version 12.1.0, everything works fine, after some upgrades when Gitlab version was 12.5.3, I find out that showing merge diffs got a 500 error. now Gitlab version is 12.6.2 and helm chart version is 2.6.3. but we still have this error.
Also, I see the same error on "background_jobs" on "new_merge_request" queue. lots of dead jobs with RuntimeError: new position is outside of file
error.
I use external object storage and it's configured correctly, I have checked that merge request diffs are stored on S3 and also when I open "changes" on merge request, a request for a diff file goes to s3 endpoint and got a 200 response code. Diff file is present on S3.
It is really strange for me where this error comes from.
Configuration used
Here is external object storage configuration
lfs:
bucket: git-lfs
connection:
secret: gitlab-rails-storage
key: connection
artifacts:
bucket: gitlab-artifacts
connection:
secret: gitlab-rails-storage
key: connection
uploads:
bucket: gitlab-uploads
connection:
secret: gitlab-rails-storage
key: connection
packages:
bucket: gitlab-packages
connection:
secret: gitlab-rails-storage
key: connection
externalDiffs:
enabled: true
when:
bucket: gitlab-mr-diffs
connection:
secret: gitlab-rails-storage
key: connection
Current behavior
stuck on loading when opening "Changes" on a merge request. Got 500 error on merge_requests/xxx/diffs.json?w=0
correctly show diffs with ?w=1 correctly show diffs when clicking on one of the commits!
Expected behavior
Show all merge request changes on "Changes" page
Relevant logs
==> /var/log/gitlab/production.log <==
RuntimeError (new position is outside of file):
lib/gitlab/http_io.rb:60:in `seek'
app/models/merge_request_diff_file.rb:20:in `block in diff'
app/models/merge_request_diff.rb:408:in `opening_external_diff'
lib/gitlab/metrics/instrumentation.rb:161:in `block in opening_external_diff'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `opening_external_diff'
app/models/merge_request_diff_file.rb:19:in `diff'
lib/gitlab/metrics/instrumentation.rb:161:in `block in diff'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `diff'
app/models/concerns/diff_file.rb:9:in `to_hash'
app/models/merge_request_diff.rb:567:in `map'
app/models/merge_request_diff.rb:567:in `block in load_diffs'
app/models/merge_request_diff.rb:413:in `block in opening_external_diff'
app/uploaders/gitlab_uploader.rb:96:in `open'
app/models/merge_request_diff.rb:410:in `opening_external_diff'
lib/gitlab/metrics/instrumentation.rb:161:in `block in opening_external_diff'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `opening_external_diff'
app/models/merge_request_diff.rb:560:in `load_diffs'
lib/gitlab/metrics/instrumentation.rb:161:in `block in load_diffs'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `load_diffs'
app/models/merge_request_diff.rb:214:in `raw_diffs'
lib/gitlab/metrics/instrumentation.rb:161:in `block in raw_diffs'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `raw_diffs'
lib/gitlab/diff/file_collection/base.rb:30:in `diffs'
lib/gitlab/diff/file_collection/base.rb:38:in `raw_diff_files'
lib/gitlab/diff/file_collection/base.rb:34:in `diff_files'
lib/gitlab/diff/file_collection/merge_request_diff_base.rb:21:in `block in diff_files'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
lib/gitlab/diff/file_collection/merge_request_diff_base.rb:20:in `diff_files'
lib/gitlab/diff/file_collection/base.rb:57:in `unfold_diff_files'
app/controllers/projects/merge_requests/diffs_controller.rb:56:in `render_diffs'
app/controllers/projects/merge_requests/diffs_controller.rb:13:in `show'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:458:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:452:in `set_locale'
lib/gitlab/error_tracking.rb:34:in `with_context'
app/controllers/application_controller.rb:536:in `sentry_context'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:62:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
ee/lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:48:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:32:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Started GET "xxxxxx/noteable/merge_request/585/notes" for xxxxxxxxx at 2020-01-04 09:42:24 +0000
Started GET "/-/readiness" for 10.44.0.0 at 2020-01-04 09:42:24 +0000
Processing by HealthController#readiness as HTML
Completed 200 OK in 0ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)