MR Diff REST API - creating note causes 500 and shows strange web UI

Summary

I found a strange behaviour that I can't reproduce on my local GitLab instance. Creating a note through GitLab API on gitlab.com causes 500 and shows a strange comment.

In overview

In diff view

Steps to reproduce

Run the following command:

curl  --request POST --header "Authorization: Bearer TOKEN" --form "body=Via API"   --form "position[new_path]=test.js"   --form "position[old_path]=test.js"  --form "position[old_line]=20" --form "position[new_line]=20"   --form "position[base_sha]=f36498b3f5ee3e31001774ee639accc7e0b8242c"   --form "position[head_sha]=f36498b3f5ee3e31001774ee639accc7e0b8242c"   --form "position[start_sha]=1f0fa02de1f6b913d674a8be10899fb8540237a9"   --form "position[position_type]=text"   https://gitlab.com/api/v4/projects/20486274/merge_requests/1/discussions

Which returns {"message":"500 Internal Server Error"}

UPDATE: I've noticed that I've put in base_sha

  "head_commit_sha": "f36498b3f5ee3e31001774ee639accc7e0b8242c",
- "base_commit_sha": "1f0fa02de1f6b913d674a8be10899fb8540237a9",
+ "base_commit_sha": "f36498b3f5ee3e31001774ee639accc7e0b8242c",
  "start_commit_sha": "1f0fa02de1f6b913d674a8be10899fb8540237a9",

With correctly filled in SHA, I get 400

The SHA was taken by running

curl -H "Authorization: Bearer TOKEN" 'https://gitlab.com/api/v4/projects/20486274/merge_requests/1/versions' | jq

And using the last version

Example Project

MR: viktomas/test-project!1 (closed)

What is the current bug behavior?

API call returns 500 but some invalid data is created.

What is the expected correct behavior?

400 error is returned without creating any data in GitLab

Relevant logs and/or screenshots

I cannot reproduce the issue locally, but this is the error in production logs:

message: Failed to find diff line for: test.js, old_line: 20, new_line: 20
class: DiffNote::NoteDiffFileCreationError
backtrace:

app/models/diff_note.rb:46:in `create_diff_file',
lib/gitlab/metrics/instrumentation.rb:160:in `block in create_diff_file',
lib/gitlab/metrics/method_call.rb:27:in `measure',
lib/gitlab/metrics/instrumentation.rb:160:in `create_diff_file',
config/initializers/forbid_sidekiq_in_transactions.rb:50:in `block in committed!',
config/initializers/forbid_sidekiq_in_transactions.rb:8:in `skipping_transaction_check',
config/initializers/forbid_sidekiq_in_transactions.rb:50:in `committed!',
ee/lib/gitlab/database/load_balancing/connection_proxy.rb:88:in `block in write_using_load_balancer',
ee/lib/gitlab/database/load_balancing/load_balancer.rb:84:in `block in read_write',
ee/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `retry_with_backoff',
ee/lib/gitlab/database/load_balancing/load_balancer.rb:83:in `read_write',
ee/lib/gitlab/database/load_balancing/connection_proxy.rb:82:in `write_using_load_balancer',
ee/lib/gitlab/database/load_balancing/connection_proxy.rb:56:in `block (2 levels) in <class:ConnectionProxy>',
app/services/notes/create_service.rb:29:in `block in execute',
app/services/notes/create_service.rb:48:in `execute_quick_actions',
app/services/notes/create_service.rb:23:in `execute',
lib/api/helpers/notes_helpers.rb:129:in `create_note',
lib/api/discussions.rb:108:in `block (3 levels) in <class:Discussions>',
ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call',
ee/lib/gitlab/ip_address_state.rb:10:in `with',
ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call',
lib/api/api_guard.rb:208:in `call',
ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase',
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call',
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call',
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call',
lib/gitlab/metrics/transaction.rb:61:in `run',
lib/gitlab/metrics/rack_middleware.rb:16:in `call',
lib/gitlab/request_profiler/middleware.rb:17:in `call',
ee/lib/gitlab/database/load_balancing/rack_middleware.rb:39:in `call',
lib/gitlab/jira/middleware.rb:19:in `call',
lib/gitlab/middleware/go.rb:20:in `call',
lib/gitlab/etag_caching/middleware.rb:13:in `call',
lib/gitlab/middleware/multipart.rb:234:in `call',
lib/gitlab/middleware/read_only/controller.rb:51:in `call',
lib/gitlab/middleware/read_only.rb:18:in `call',
lib/gitlab/middleware/same_site_cookies.rb:27:in `call',
lib/gitlab/middleware/handle_malformed_strings.rb:19:in `call',
lib/gitlab/middleware/basic_health_check.rb:25:in `call',
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call',
lib/gitlab/middleware/request_context.rb:23:in `call',
config/initializers/fix_local_cache_middleware.rb:9:in `call',
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call',
lib/gitlab/middleware/release_env.rb:12:in `call'

Link to our production logs (the log is going to be gone roughly a week after now=2020-11-11T09:03:47.145Z)

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes