Downloading full commit diffs is impossible using API Personal Access Tokens

Summary

Since the commits API does not provide a way to download the full diff of a commit (large diffs are truncated to empty string, as described in #57702 (moved)), the workaround we had in one of our legacy applications was to download that diff using the /<namespace>/<project>/commit/<sha>.diff route.

In v11.6.0, Personal Access Tokens became restricted to routes starting with /api and our workaround stopped working. We had to fallback to the workaround listed at the end of this issue.

Steps to reproduce

  1. create a private project on gitlab.

  2. create a Personal Access Token with 'api' scope which has access to that project.

  3. download the diff of an existing commit from the private project using the Personal Access Token

curl -sSL -H "Private-Token: XXXXX" 'https://my.gitlab.instance/mygroup/myproject/commit/e512cd4c8bcc382a1e3d91447e9cd2d049da8389.diff'

What is the current bug behavior?

Using a recent gitlab-ce (such as v11.8.1), the token is denied access to the route (http status 401). The downloaded body content is:

You need to sign in or sign up before continuing.

What is the expected correct behavior?

Using a gitlab-ce prior to v11.6.0, the token is granted access to the route (http status 200): The downloaded body content is:

+++ diff content follows

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 14.04
Current User:	git
Using RVM:	no
Ruby Version:	2.5.3p105
Gem Version:	2.7.6
Bundler Version:1.16.6
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.18.1
Sidekiq Version:5.2.5
Go Version:	unknown

GitLab information
Version:	11.8.1
Revision:	657d508
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://mingalar.fr:441
HTTP Clone URL:	https://mingalar.fr:441/some-group/some-project.git
SSH Clone URL:	ssh://git@mingalar.fr:2223/some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	8.4.4
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
DN: cn=julien perville,ou=people,dc=mingalar,dc=fr	 uid: julien
DN: cn=fabien catteau,ou=people,dc=mingalar,dc=fr	 uid: fabien
DN: cn=laurent perville,ou=people,dc=mingalar,dc=fr	 uid: laurent
DN: cn=alain chagneau,ou=people,dc=mingalar,dc=fr	 uid: alain


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
3/3 ... yes
2/4 ... yes
3/5 ... yes
4/6 ... yes
4/7 ... yes
4/8 ... yes
6/9 ... yes
7/10 ... yes
7/11 ... yes
2/12 ... yes
4/13 ... yes
4/14 ... yes
2/15 ... yes
4/16 ... yes
4/17 ... yes
7/18 ... yes
7/19 ... yes
4/20 ... yes
4/21 ... yes
12/22 ... yes
11/23 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 5


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Commit responsible from the change: https://gitlab.com/gitlab-org/gitlab-ce/commit/fe5f75930e781ef854b458fafa307ebb90a8ed2e

The change may be intentional but it leaves us without any way to download the full diff. Maybe the commit API should be extended to have that feature.

Meanwhile, our workaround is to patch the installed gitlab-ce code and run sudo gitlab-ctl restart unicorn after each installation.

Workaround patch content:

diff --git a/embedded/service/gitlab-rails/app/controllers/projects/commit_controller.rb b/embedded/service/gitlab-rails/app/controllers/projects/commit_c
index b13c0ae..1eb1e2f 100644
--- a/embedded/service/gitlab-rails/app/controllers/projects/commit_controller.rb
+++ b/embedded/service/gitlab-rails/app/controllers/projects/commit_controller.rb
@@ -10,6 +10,7 @@ class Projects::CommitController < Projects::ApplicationController
   include DiffHelper
 
   # Authorize
+  prepend_before_action(only: [:show]) { authenticate_sessionless_user!(:api) }
   before_action :require_non_empty_project
   before_action :authorize_download_code!
   before_action :authorize_read_pipeline!, only: [:pipelines]
diff --git a/embedded/service/gitlab-rails/lib/gitlab/auth/user_auth_finders.rb b/embedded/service/gitlab-rails/lib/gitlab/auth/user_auth_finders.rb
index a5efe33..e623916 100644
--- a/embedded/service/gitlab-rails/lib/gitlab/auth/user_auth_finders.rb
+++ b/embedded/service/gitlab-rails/lib/gitlab/auth/user_auth_finders.rb
@@ -152,7 +152,7 @@ module Gitlab
       end
 
       def api_request?
-        current_request.path.starts_with?("/api/")
+        current_request.path.starts_with?("/api/") || current_request.path.include?('/commit/')
       end
     end
   end