Skip to content

Retrieving diff of recently deleted MR returns a 500

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

An API request to get an MR diff (GET /api/:version/projects/:id/merge_requests/:merge_request_iid/diffs) returns a 500 if the MR was very recently deleted (DELETE /api/:version/projects/:id/merge_requests/:merge_request_iid). Subsequent calls to the same API returns the expected 404 response after some time has passed.

Steps to reproduce

We encounter this situation periodically with frequently running canaries from two overlapping features that end up doing the following:

  1. Feature A Canary: Creates an MR.
  2. Feature B Canary: Detects the new MR has been created, begins performing some work related to it.
  3. Feature A Canary: Completes its testing. Deletes MR to clean up resources used.
  4. Feature B Canary: Tries to retrieve a diff for the MR that was just deleted, gets a 500 response.
  5. Feature B Canary: Performs a retry on the same MR diff API call, gets the expected 404 response.

Example Project

Not available.

What is the current bug behavior?

Calling a GET /api/:version/projects/:id/merge_requests/:merge_request_iid/diffs for an MR that was deleted within a second of a DELETE /api/:version/projects/:id/merge_requests/:merge_request_iid call returns an unexpected 500 response.

What is the expected correct behavior?

The GET /api/:version/projects/:id/merge_requests/:merge_request_iid/diffs request should return a 404.

Relevant logs and/or screenshots

Note that the GET that returns a 500 and the DELETE show as starting at the same time in the logs. I suspect the issue is happening when requests are overlapping in this way.

gitlab-rails/api_json.log

@timestamp severity duration_s status method path
2025-04-25 14:53:09.017 INFO 0.0698 404 GET /api/v4/projects/21/merge_requests/454523/diffs
2025-04-25 14:52:18.147 INFO 0.06286 404 GET /api/v4/projects/21/merge_requests/454523/diffs
2025-04-25 14:52:07.126 INFO 0.05128 404 GET /api/v4/projects/21/merge_requests/454523/diffs
2025-04-25 14:52:05.622 INFO 0.21384 204 DELETE /api/v4/projects/21/merge_requests/454523
2025-04-25 14:52:05.622 INFO 0.16735 500 GET /api/v4/projects/21/merge_requests/454523/diffs
2025-04-25 14:52:02.113 INFO 0.16227 200 GET /api/v4/projects/21/merge_requests/454523

For every instance of a 500 where we see this, the exception fields logged are:

  • exception.class: NoMethodError
  • exception.message: undefined method ``target_project' for nil
  • exception.backtrace:
[
    "app/models/merge_request_diff.rb:536:in `project'",
    "lib/gitlab/diff/file_collection/merge_request_diff_base.rb:20:in `initialize'",
    "lib/gitlab/diff/file_collection/paginated_merge_request_diff.rb:19:in `initialize'",
    "app/models/merge_request_diff.rb:493:in `new'",
    "app/models/merge_request_diff.rb:493:in `block in paginated_diffs'",
    "app/models/merge_request_diff.rb:763:in `fetching_repository_diffs'",
    "app/models/merge_request_diff.rb:490:in `paginated_diffs'",
    "lib/api/merge_requests.rb:606:in `block (2 levels) in %3Cclass:MergeRequests%3E'",
    "grape (2.0.0) lib/grape/endpoint.rb:58:in `call'",
    "grape (2.0.0) lib/grape/endpoint.rb:58:in `block (2 levels) in generate_api_method'",
    "activesupport (7.0.8.7) lib/active_support/notifications.rb:208:in `instrument'",
    "grape (2.0.0) lib/grape/endpoint.rb:57:in `block in generate_api_method'",
    "grape (2.0.0) lib/grape/endpoint.rb:328:in `execute'",
    "grape (2.0.0) lib/grape/endpoint.rb:260:in `block in run'",
    "activesupport (7.0.8.7) lib/active_support/notifications.rb:208:in `instrument'",
    "grape (2.0.0) lib/grape/endpoint.rb:240:in `run'",
    "grape (2.0.0) lib/grape/endpoint.rb:316:in `block in build_stack'",
    "grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "lib/api/api_guard.rb:247:in `call'",
    "grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource.rb:20:in `_call'",
    "rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'",
    "rack-oauth2 (2.2.1) lib/rack/oauth2/server/abstract/handler.rb:17:in `call'",
    "ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'",
    "lib/gitlab/ip_address_state.rb:11:in `with'",
    "ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'",
    "grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "lib/api/api_guard.rb:247:in `call'",
    "grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource.rb:20:in `_call'",
    "rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'",
    "rack-oauth2 (2.2.1) lib/rack/oauth2/server/abstract/handler.rb:17:in `call'",
    "grape (2.0.0) lib/grape/middleware/error.rb:39:in `block in call!'",
    "grape (2.0.0) lib/grape/middleware/error.rb:38:in `catch'",
    "grape (2.0.0) lib/grape/middleware/error.rb:38:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'",
    "grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:58:in `catch'",
    "grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:58:in `call!'",
    "grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
    "rack (2.2.13) lib/rack/head.rb:12:in `call'",
    "grape (2.0.0) lib/grape/endpoint.rb:224:in `call!'",
    "grape (2.0.0) lib/grape/endpoint.rb:218:in `call'",
    "grape (2.0.0) lib/grape/router/route.rb:58:in `exec'",
    "grape (2.0.0) lib/grape/router.rb:120:in `process_route'",
    "grape (2.0.0) lib/grape/router.rb:74:in `block in identity'",
    "grape (2.0.0) lib/grape/router.rb:94:in `transaction'",
    "grape (2.0.0) lib/grape/router.rb:72:in `identity'",
    "grape (2.0.0) lib/grape/router.rb:56:in `block in call'",
    "grape (2.0.0) lib/grape/router.rb:136:in `with_optimization'",
    "grape (2.0.0) lib/grape/router.rb:55:in `call'",
    "grape (2.0.0) lib/grape/api/instance.rb:165:in `call'",
    "grape (2.0.0) lib/grape/api/instance.rb:70:in `call!'",
    "grape (2.0.0) lib/grape/api/instance.rb:65:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'",
    "actionpack (7.0.8.7) lib/action_dispatch/routing/mapper.rb:48:in `serve'",
    "actionpack (7.0.8.7) lib/action_dispatch/journey/router.rb:50:in `block in serve'",
    "actionpack (7.0.8.7) lib/action_dispatch/journey/router.rb:32:in `each'",
    "actionpack (7.0.8.7) lib/action_dispatch/journey/router.rb:32:in `serve'",
    "actionpack (7.0.8.7) lib/action_dispatch/routing/route_set.rb:852:in `call'",
    "gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'",
    "flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'",
    "flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call'",
    "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
    "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
    "lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
    "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
    "lib/gitlab/middleware/memory_report.rb:13:in `call'",
    "lib/gitlab/middleware/speedscope.rb:13:in `call'",
    "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
    "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
    "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
    "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
    "lib/gitlab/metrics/web_transaction.rb:46:in `run'",
    "lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
    "lib/gitlab/middleware/go.rb:21:in `call'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
    "lib/gitlab/database/query_analyzer.rb:83:in `within'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
    "lib/ci/job_token/middleware.rb:11:in `call'",
    "batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'",
    "rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'",
    "apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'",
    "lib/gitlab/middleware/multipart.rb:173:in `call'",
    "rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'",
    "warden (1.2.9) lib/warden/manager.rb:36:in `block in call'",
    "warden (1.2.9) lib/warden/manager.rb:34:in `catch'",
    "warden (1.2.9) lib/warden/manager.rb:34:in `call'",
    "rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'",
    "rack (2.2.13) lib/rack/tempfile_reaper.rb:15:in `call'",
    "rack (2.2.13) lib/rack/etag.rb:27:in `call'",
    "rack (2.2.13) lib/rack/conditional_get.rb:27:in `call'",
    "rack (2.2.13) lib/rack/head.rb:12:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/http/permissions_policy.rb:38:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/http/content_security_policy.rb:39:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
    "rack (2.2.13) lib/rack/session/abstract/id.rb:266:in `context'",
    "rack (2.2.13) lib/rack/session/abstract/id.rb:260:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/cookies.rb:704:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'",
    "activesupport (7.0.8.7) lib/active_support/callbacks.rb:99:in `run_callbacks'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/callbacks.rb:26:in `call'",
    "sentry-rails (5.22.1) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'",
    "lib/gitlab/middleware/path_traversal_check.rb:35:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "sentry-ruby (5.22.1) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'",
    "sentry-ruby (5.22.1) lib/sentry/hub.rb:269:in `with_session_tracking'",
    "sentry-ruby (5.22.1) lib/sentry-ruby.rb:416:in `with_session_tracking'",
    "sentry-ruby (5.22.1) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'",
    "sentry-ruby (5.22.1) lib/sentry/hub.rb:59:in `with_scope'",
    "sentry-ruby (5.22.1) lib/sentry-ruby.rb:396:in `with_scope'",
    "sentry-ruby (5.22.1) lib/sentry/rack/capture_exceptions.rb:20:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'",
    "railties (7.0.8.7) lib/rails/rack/logger.rb:25:in `block in call'",
    "activesupport (7.0.8.7) lib/active_support/tagged_logging.rb:99:in `block in tagged'",
    "activesupport (7.0.8.7) lib/active_support/tagged_logging.rb:37:in `tagged'",
    "activesupport (7.0.8.7) lib/active_support/tagged_logging.rb:99:in `tagged'",
    "railties (7.0.8.7) lib/rails/rack/logger.rb:25:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:15:in `call'",
    "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
    "request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'",
    "rack (2.2.13) lib/rack/method_override.rb:24:in `call'",
    "rack (2.2.13) lib/rack/runtime.rb:22:in `call'",
    "rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'",
    "rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'",
    "rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/middleware/compressed_json.rb:44:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/executor.rb:14:in `call'",
    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
    "rack (2.2.13) lib/rack/sendfile.rb:110:in `call'",
    "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
    "gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:22:in `block in call'",
    "gitlab-labkit (0.37.0) lib/labkit/context.rb:35:in `with_context'",
    "gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:21:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/request_id.rb:26:in `call'",
    "actionpack (7.0.8.7) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'",
    "railties (7.0.8.7) lib/rails/engine.rb:530:in `call'",
    "railties (7.0.8.7) lib/rails/railtie.rb:226:in `public_send'",
    "railties (7.0.8.7) lib/rails/railtie.rb:226:in `method_missing'",
    "lib/gitlab/middleware/release_env.rb:12:in `call'",
    "rack (2.2.13) lib/rack/urlmap.rb:74:in `block in call'",
    "rack (2.2.13) lib/rack/urlmap.rb:58:in `each'",
    "rack (2.2.13) lib/rack/urlmap.rb:58:in `call'",
    "puma (6.5.0) lib/puma/configuration.rb:279:in `call'",
    "puma (6.5.0) lib/puma/request.rb:99:in `block in handle_request'",
    "puma (6.5.0) lib/puma/thread_pool.rb:389:in `with_force_shutdown'",
    "puma (6.5.0) lib/puma/request.rb:98:in `handle_request'",
    "puma (6.5.0) lib/puma/server.rb:468:in `process_client'",
    "puma (6.5.0) lib/puma/server.rb:249:in `block in run'",
    "puma (6.5.0) lib/puma/thread_pool.rb:166:in `block in spawn_thread'"
]

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
/opt/gitlab/embedded/lib/ruby/gems/3.3.0/gems/fast_gettext-2.3.0/lib/fast_gettext/vendor/mofile.rb:24: warning: prime was loaded from the standard library, but is not part of the default gems starting from Ruby 3.1.0.
You can add prime to your Gemfile or gemspec to silence this warning.

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.3.7
Gem Version:    3.6.7
Bundler Version:2.6.5
Rake Version:   13.0.6
Redis Version:  7.2.7
Sidekiq Version:7.3.9
Go Version:     unknown

GitLab information
Version:        17.11.0-pre
Revision:       1bcb23968e5
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     16.8
URL:            https://{redacted}
HTTP Clone URL: https://{redacted}/some-group/some-project.git
SSH Clone URL:  git@e2e.gamma.dev.us-east-1.gitlab.falco.ai.aws.dev:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.41.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.10.0-rc1-197-g58bf41c6a
- default Git Version:  2.49.0.gl1

Results of GitLab application Check

Expand for output related to the GitLab application check 
/opt/gitlab/embedded/lib/ruby/gems/3.3.0/gems/fast_gettext-2.3.0/lib/fast_gettext/vendor/mofile.rb:24: warning: prime was loaded from the standard library, but is not part of the default gems starting from Ruby 3.1.0.
You can add prime to your Gemfile or gemspec to silence this warning.
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.41.0 ? ... OK (14.41.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
3/1 ... yes
13/4 ... yes
19/5 ... yes
11/6 ... yes
19/7 ... yes
19/8 ... yes
19/9 ... yes
31/10 ... yes
31/11 ... yes
37/12 ... yes
40/13 ... yes
39/15 ... yes
19/16 ... yes
50/17 ... yes
29/18 ... yes
57/20 ... yes
57/21 ... yes
57/22 ... yes
13/23 ... yes
57/24 ... yes
44/25 ... yes
78/26 ... yes
19/27 ... yes
13/28 ... yes
57/29 ... yes
13/30 ... yes
13/31 ... yes
13/32 ... yes
13/33 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.3.7)
Git user has default SSH configuration? ... yes
Active users: ... 42
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Edited by 🤖 GitLab Bot 🤖