Merge Request Diffs not showing, controller reporting 404

Summary

When clicking on the first Compare with previous version in a merge request, the diff page opens but doesn't show anything great but a message that something went wrong internally.

Explanation and the current bug behavior

I'm experiencing a similar problem as in #23841 (closed), but the diff I expect to get is not empty. Howver there is an 404 answer.

This is in a private selfhosted project. So I can only describe a bit what I observed:

The MR is for a feature branch to be merged in master. I think it has been created from the issue page, howeber this is not 100% certain, but this is as we usually work.

The first comment (leaving alone some label changes) is:

image

The link d93328f1...a1b63653 works and results in the diff disabled. The link is https://hostname/p2/std/-/compare/d93328f1dd9df47bc4112269facf6fdf78de01bd...a1b63653e0abc55414c7526907db0ed3d8464e42.

But clicking on the Compare with previous version link, opens new page with the waiting symbol turning and turning and telling me that something went wrong:

image

The link is pointing to https://hostname/p2/std/-/merge_requests/177/diffs?diff_id=551&start_sha=d93328f1dd9df47bc4112269facf6fdf78de01bd.

The start_sha is valid, as it is the same which is used for the working comparison.

(What actually happens)

What is the expected correct behavior?

I'm not sure, but I would expect the diff d93328f1...a1b63653 to show up, as that is working in the comment above.

However while investigating, I found that pushes of which the Compare with previous version link is not working, are only importing (merging in) commits from the MR target branch. I'm not sure what is the intended function of the link in this case.

But in any case, there should be more than just the message that something went wrong internally and a stuck diff page:

  • If the intended function is showing the same diff as d93328f1...a1b63653, then there is a bug.
  • If it is intended that the diff is empty, then either the link on the MR page shouldn't be shown, or at least there should be an explanation why that diff is empty.

Relevant logs and/or screenshots

The logs (that I think are related to the operation) show this:

{"method":"GET","path":"/p2/std/-/merge_requests/177/diffs","format":"html","controller":"Projects::MergeRequestsController","action":"show","status":200,"time":"2020-06-12T11:56:32.230Z","params":[{"key":"diff_id","value":"551"},{"key":"start_sha","value":"d93328f1dd9df47bc4112269facf6fdf78de01bd"},{"key":"tab","value":"diffs"},{"key":"namespace_id","value":"p2"},{"key":"project_id","value":"std"},{"key":"id","value":"177"}],"remote_ip":"0.0.0.0","user_id":2,"username":"js","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0","queue_duration_s":0.005147,"correlation_id":"Rg2HsdItZX5","meta.user":"js","meta.project":"p2/std","meta.root_namespace":"p2","meta.caller_id":"Projects::MergeRequestsController#show","redis_calls":33,"redis_duration_s":0.011005,"cpu_s":0.5,"db_duration_s":0.06919,"view_duration_s":0.34503,"duration_s":0.53617}
{"method":"GET","path":"/p2/std/-/merge_requests/177/diffs_metadata.json","format":"json","controller":"Projects::MergeRequests::DiffsController","action":"diffs_metadata","status":404,"time":"2020-06-12T11:56:33.892Z","params":[{"key":"diff_id","value":"551"},{"key":"start_sha","value":"d93328f1dd9df47bc4112269facf6fdf78de01bd"},{"key":"view","value":"inline"},{"key":"namespace_id","value":"p2"},{"key":"project_id","value":"std"},{"key":"id","value":"177"}],"remote_ip":"0.0.0.0","user_id":2,"username":"js","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0","queue_duration_s":0.007161,"correlation_id":"PQoADpKdSd4","meta.user":"js","meta.project":"p2/std","meta.root_namespace":"p2","meta.caller_id":"Projects::MergeRequests::DiffsController#diffs_metadata","redis_calls":2,"redis_duration_s":0.001286,"cpu_s":0.07,"db_duration_s":0.01842,"view_duration_s":0.0,"duration_s":0.06539}
{"method":"GET","path":"/p2/std/-/merge_requests/177/diffs_batch.json","format":"json","controller":"Projects::MergeRequests::DiffsController","action":"diffs_batch","status":404,"time":"2020-06-12T11:56:33.904Z","params":[{"key":"diff_id","value":"551"},{"key":"start_sha","value":"d93328f1dd9df47bc4112269facf6fdf78de01bd"},{"key":"per_page","value":"20"},{"key":"w","value":"0"},{"key":"view","value":"inline"},{"key":"page","value":"1"},{"key":"namespace_id","value":"p2"},{"key":"project_id","value":"std"},{"key":"id","value":"177"}],"remote_ip":"0.0.0.0","user_id":2,"username":"js","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0","queue_duration_s":0.006951,"correlation_id":"4Y7Rtda1Ot6","meta.user":"js","meta.project":"p2/std","meta.root_namespace":"p2","meta.caller_id":"Projects::MergeRequests::DiffsController#diffs_batch","redis_calls":2,"redis_duration_s":0.001153,"cpu_s":0.07,"db_duration_s":0.01277,"view_duration_s":0.0,"duration_s":0.07264}
{"method":"GET","path":"/p2/std/-/merge_requests/177.json","format":"json","controller":"Projects::MergeRequestsController","action":"show","status":200,"time":"2020-06-12T11:56:33.962Z","params":[{"key":"serializer","value":"sidebar_extras"},{"key":"namespace_id","value":"p2"},{"key":"project_id","value":"std"},{"key":"id","value":"177"}],"remote_ip":"0.0.0.0","user_id":2,"username":"js","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0","queue_duration_s":0.005006,"correlation_id":"4K4Y0PXJAL6","meta.user":"js","meta.project":"p2/std","meta.root_namespace":"p2","meta.caller_id":"Projects::MergeRequestsController#show","redis_calls":1,"redis_duration_s":0.000218,"cpu_s":0.13,"db_duration_s":0.02493,"view_duration_s":0.00073,"duration_s":0.1372}

I tried to find out about the root cause and looked into the database:

gitlabhq_production=> select id, iid from merge_requests where iid=177;
 id  | iid
-----+-----
 195 | 177
(1 row)

select * from merge_request_diffs where merge_request_id=195 order by id;
 id  |   state   | merge_request_id |         created_at         |         updated_at         |             base_commit_sha              | real_size |             head_commit_sha              |             start_commit_sha             | commits_count | external_diff | external_diff_store | stored_externally
-----+-----------+------------------+----------------------------+----------------------------+------------------------------------------+-----------+------------------------------------------+------------------------------------------+---------------+---------------+---------------------+-------------------
 544 | empty     |              195 | 2020-02-06 12:14:25.037922 | 2020-02-06 12:14:25.140733 | d93328f1dd9df47bc4112269facf6fdf78de01bd |           | d93328f1dd9df47bc4112269facf6fdf78de01bd | d93328f1dd9df47bc4112269facf6fdf78de01bd |             0 | diff-544      |                   1 |
 551 | empty     |              195 | 2020-03-03 10:46:33.99659  | 2020-03-03 10:46:34.14423  | a1b63653e0abc55414c7526907db0ed3d8464e42 |           | a1b63653e0abc55414c7526907db0ed3d8464e42 | a1b63653e0abc55414c7526907db0ed3d8464e42 |             0 | diff-551      |                   1 |
 554 | empty     |              195 | 2020-03-09 07:11:01.270213 | 2020-03-09 07:11:01.462231 | c9dde4b01d2f2030a8b0820b0dd1739888c934cd |           | c9dde4b01d2f2030a8b0820b0dd1739888c934cd | c9dde4b01d2f2030a8b0820b0dd1739888c934cd |             0 | diff-554      |                   1 |
 556 | collected |              195 | 2020-03-18 08:14:31.150774 | 2020-03-18 08:14:32.102089 | c9dde4b01d2f2030a8b0820b0dd1739888c934cd | 34        | 6791c3f052a8dd62630ea93240566fb089fca7a8 | c9dde4b01d2f2030a8b0820b0dd1739888c934cd |             1 | diff-556      |                   1 |
 557 | collected |              195 | 2020-03-19 08:34:31.684547 | 2020-03-19 08:34:31.997448 | 04d1aa85b358a25135b96bd129485c4b28db60e2 | 34        | 1019ebaf6770b6d0efb5d5c3fc4969c2045cacca | 04d1aa85b358a25135b96bd129485c4b28db60e2 |             2 | diff-557      |                   1 |
 558 | collected |              195 | 2020-03-23 09:31:01.630583 | 2020-03-23 09:31:02.060449 | 04d1aa85b358a25135b96bd129485c4b28db60e2 | 40        | a475a1ac59bafa77fe24925639d3bc025c4c191c | 78175dfbf6c1d39af4c7aa9c40dc0b42590eec01 |             3 | diff-558      |                   1 |
 559 | collected |              195 | 2020-03-23 13:46:18.133881 | 2020-03-23 13:46:18.61507  | 3051aff5d204bd4ff6919ebf081842e51f48fd73 | 40        | bd542ce6fe485292d1cc0c26779cd9600ac280fc | 3051aff5d204bd4ff6919ebf081842e51f48fd73 |             4 | diff-559      |                   1 |
 577 | collected |              195 | 2020-06-03 05:56:24.66089  | 2020-06-03 05:56:25.083831 | 3051aff5d204bd4ff6919ebf081842e51f48fd73 | 45        | 1adcdd6579672eeb5e300afc4563847c1f5be911 | e4248d4c2d0c3364864224529331a2d6ccc43342 |             5 | diff-577      |                   1 |
 578 | collected |              195 | 2020-06-03 06:55:03.913111 | 2020-06-03 06:55:04.392354 | e4248d4c2d0c3364864224529331a2d6ccc43342 | 45        | c12bde31ddd69693ce0422889b7a6f978970c592 | e4248d4c2d0c3364864224529331a2d6ccc43342 |             6 | diff-578      |                   1 |
 579 | collected |              195 | 2020-06-08 13:46:46.549437 | 2020-06-08 13:46:47.326988 | e4248d4c2d0c3364864224529331a2d6ccc43342 | 45        | 29c3ad9a36fece500e1c76683a72dc8d2545f6f8 | e4248d4c2d0c3364864224529331a2d6ccc43342 |             7 | diff-579      |                   1 |
(10 rows)

gitlabhq_production=> select * from merge_request_diff_commits where merge_request_diff_id in (554, 551, 554, 556) order by merge_request_diff_id, relative_order;
    authored_date    |   committed_date    | merge_request_diff_id | relative_order |                    sha                     |   author_name   |   author_email    | committer_name  |  committer_email  |                                   message
---------------------+---------------------+-----------------------+----------------+--------------------------------------------+-----------------+-------------------+-----------------+-------------------+--------------------------------------------------------------------------------------------------------
 2020-03-18 08:14:23 | 2020-03-18 08:14:23 |                   556 |              0 | \x6791c3f052a8dd62630ea93240566fb089fca7a8 | Xxxxxxxxxxxxxxx | xxxxxxxxxxxxxxxxx | Xxxxxxxxxxxxxxx | xxxxxxxxxxxxxxxxx | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX                                                                        +

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Ubuntu 18.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.6.6p146
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.9
Git Version:    2.26.2
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        13.0.5-ee
Revision:       260c6231ed0
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.7
URL:            https://hostname
HTTP Clone URL: https://hostname/some-group/some-project.git
SSH Clone URL:  git@hostname:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.2.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.2.0 ? ... OK (13.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server 'demig.intra:636'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 19 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
2/3 ... yes
6/5 ... yes
6/6 ... yes
11/7 ... yes
3/8 ... yes
12/9 ... yes
6/10 ... yes
13/11 ... yes
13/12 ... yes
13/13 ... yes
13/14 ... yes
3/15 ... yes
3/16 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 8
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)