Blank raw files

Summary

When I try to view/download a raw file in a repository it shows up blank.

Steps to reproduce

  1. Install an Ubuntu 20.04 Server OS. (I'm using Oracle Cloud Infrastructure Free Tier - Arm64 VM with 24G RAM [Neofetch Provided])
  2. Install GitLab CE (Omnibus) 14.15.1-ce.0 (On my server: "gitlab-ce/focal,now 14.5.1-ce.0 arm64" [apt list --installed | grep gitlab]) [It was just upgraded]
  3. Configure it so that it works behind an Apache2 reverse proxy. (Configuration provided, I originally used this tutorial)
  4. Uninstall UFW and Fail2Ban
  5. Install Firewalld
  6. Allow the required ports
  7. Create an account and a repository.
  8. Upload a file.
  9. Open the raw version or try to download it.

Example Project

This is on a self-hosted instance. The raw url is something like: https://git.mydomain.com/example-group/example-project/-/raw/main/build.gradle.kts

What is the current bug behavior?

It just displays a blank file.

What is the expected correct behavior?

It would show the file contents.

Relevant logs and/or screenshots

GitLab Rails Logs 
ubuntu@my-vm:~$ sudo tail -f /var/log/gitlab/gitlab-rails/production.log
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:51:44 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:51:47 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:51:50 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:51:53 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:51:56 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:51:59 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:02 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:05 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:08 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:11 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:14 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:17 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:20 +0000
Started GET "/example-group/example-project/-/raw/main/build.gradle.kts" for 76.115.32.219 at 2021-12-06 05:52:22 +0000
Processing by Projects::RawController#show as HTML
  Parameters: {"namespace_id"=>"example-group", "project_id"=>"example-project", "id"=>"main/build.gradle.kts"}
Completed 200 OK in 213ms (Views: 0.6ms | ActiveRecord: 6.4ms | Elasticsearch: 0.0ms | Allocations: 13436)
Started GET "/favicon.ico" for ***.***.***.*** at 2021-12-06 05:52:22 +0000
Started POST "/api/v4/jobs/request" for ***.***.***.*** at 2021-12-06 05:52:23 +0000
^C
ubuntu@my-vm:~$
Neofetch 
            .-/+oossssoo+/-.               ubuntu@my-vm
        `:+ssssssssssssssssss+:`           -----------------------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.3 LTS aarch64
    .ossssssssssssssssssdMMMNysssso.       Host: KVM Virtual Machine virt-4.2
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.11.0-1022-oracle
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 10 hours, 18 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 2751 (dpkg), 14 (snap)
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.0.17
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1024x768
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Terminal: /dev/pts/9
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: (4)
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: 00:01.0 Red Hat, Inc. Virtio GPU
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Memory: 18940MiB / 23872MiB
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.
image
Response Headers 
HTTP/1.1 200 OK
Date: Mon, 06 Dec 2021 05:52:22 GMT
Server: Apache/2.4.41 (Ubuntu)
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
X-UA-Compatible: IE=edge
ETag: W/"958e1f77bca635952de6c782ef803347-gzip"
Gitlab-Workhorse-Send-Data: git-blob:eyJHaXRhbHlTZXJ2ZXIiOnsiYWRkcmVzcyI6InVuaXg6L3Zhci9vcHQvZ2l0bGFiL2dpdGFseS9naXRhbHkuc29ja2V0IiwidG9rZW4iOiIiLCJmZWF0dXJlcyI6e319LCJHZXRCbG9iUmVxdWVzdCI6eyJyZXBvc2l0b3J5Ijp7InN0b3JhZ2VfbmFtZSI6ImRlZmF1bHQiLCJyZWxhdGl2ZV9wYXRoIjoiQGhhc2hlZC80Yi8yMi80YjIyNzc3N2Q0ZGQxZmM2MWM2Zjg4NGY0ODY0MWQwMmI0ZDEyMWQzZmQzMjhjYjA4YjU1MzFmY2FjZGFiZjhhLmdpdCIsImdpdF9vYmplY3RfZGlyZWN0b3J5IjoiIiwiZ2l0X2FsdGVybmF0ZV9vYmplY3RfZGlyZWN0b3JpZXMiOltdLCJnbF9yZXBvc2l0b3J5IjoicHJvamVjdC00IiwiZ2xfcHJvamVjdF9wYXRoIjoibm9zYWRuaWxlLW5ldHdvcmsvbm9zYWRuaWxlIn0sIm9pZCI6IjEzNDgxNjRjMWEzZGZmYmQ3NzY2OTZiMzhmOWI4NjE4ZWU5OWU1ZTUiLCJsaW1pdCI6LTF9fQ==
Content-Disposition: inline
Gitlab-Workhorse-Detect-Content-Type: true
Content-Type: text/plain; charset=utf-8
Permissions-Policy: interest-cohort=()
Cache-Control: max-age=60, public
X-Runtime: 0.241657
X-Request-Id: 9262058a-6bc6-4916-bd28-543b193cb843
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Apache2 configuration file (/etc/apache2/sites-available/gitlab.conf) 
<VirtualHost *:443>
  ServerName git.mydomain.com
  ServerSignature Off

  ProxyPreserveHost On
  AllowEncodedSlashes NoDecode


  <Location />
    Require all granted
    ProxyPassReverse http://127.0.0.1:9988
    ProxyPassReverse git.mydomain.com
  </Location>


  RewriteEngine on
  RewriteRule /[-/\w.]+.git/ http://127.0.0.1:8282%{REQUEST_URI} [P,QSA,L]
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule .* http://127.0.0.1:9988%{REQUEST_URI} [P,QSA,NE]


  RequestHeader set X-Forwarded-Proto "https"
  RequestHeader set X-Forwarded-Ssl "on"


  DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public


  ErrorDocument 404 /404.html
  ErrorDocument 422 /422.html
  ErrorDocument 500 /500.html
  ErrorDocument 503 /deploy.html


  LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b" common_forwarded
  ErrorLog  /var/log/httpd/logs/git.mydomain.com_error.log
  CustomLog /var/log/httpd/logs/git.mydomain.com_forwarded.log common_forwarded
  CustomLog /var/log/httpd/logs/git.mydomain.com_access.log combined env=!dontlog
  CustomLog /var/log/httpd/logs/git.mydomain.com.log combined


  SSLCertificateFile /etc/letsencrypt/live/git.mydomain.com/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/git.mydomain.com/privkey.pem
  Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
<VirtualHost *:80>
  ServerName git.mydomain.com
  DocumentRoot /var/www/html
  RewriteEngine on
  RewriteCond %{SERVER_NAME} =git.mydomain.com
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
Firewalld open ports 
ubuntu@my-vm:~$ sudo firewall-cmd --list-all
You're performing an operation over default zone ('public'),
but your connections/interfaces are in zone 'docker' (see --get-active-zones)
You most likely need to use --zone=docker option.

public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: dhcpv6-client ssh
ports: 90/tcp 90/udp 80/tcp 80/udp 443/tcp 443/udp 8080/tcp 8080/udp 5800/tcp 5800/udp 5900/tcp 5900/udp 5901/tcp 5901/udp 6379/tcp 6379/udp 2022/tcp 2022/udp 2000/tcp 2000/udp 2001/tcp 2001/udp 3306/tcp 3306/udp 25587/tcp 25587/udp 6566/tcp 6566/udp 25565/tcp 25565/udp 19132/tcp 19132/udp 3333/tcp 3333/udp 193/tcp 193/udp 1935/tcp 1935/udp 8888/tcp 8888/udp 9800/tcp 9800/udp 27017/tcp 27017/udp 9099/tcp 9099/udp 4000/tcp 4000/udp 9090/tcp 9090/udp 9226/tcp 9226/udp 2002/tcp 2002/udp 5500/tcp 5500/udp 1936/tcp 1936/udp 8000/tcp 8000/udp 5000/tcp 5000/udp 6000/tcp 6000/udp 5001/tcp 5001/udp 6901/tcp 6901/udp 3100/tcp 3100/udp 4200/tcp 4200/udp 1447/tcp 1447/udp 8090/tcp 8090/udp 53/tcp 53/udp 52709/tcp 52709/udp 9080/tcp 9080/udp 9091/tcp 9091/udp 2015/tcp 2015/udp 9092/tcp 9092/udp 8981/tcp 8981/udp 8980/tcp 8980/udp 8982/tcp 8982/udp 8983/tcp 8983/udp 5080/tcp 5080/udp 5081/tcp 5081/udp 5090/tcp 5090/udp 8001/tcp 8001/udp 8002/tcp 8002/udp 8003/tcp 8003/udp 8004/tcp 8004/udp 8005/tcp 8005/udp 8006/tcp 8006/udp 8007/tcp 8007/udp 8008/tcp 8008/udp 8009/tcp 8009/udp 8010/tcp 8010/udp 8011/tcp 8011/udp 8012/tcp 8012/udp 8013/tcp 8013/udp 8014/tcp 8014/udp 6443/tcp 6443/udp 10250/tcp 10250/udp 32518/tcp 32518/udp 32000/tcp 32000/udp 8900/tcp 8900/udp 8081/tcp 8081/udp 8082/tcp 8082/udp 8083/tcp 8083/udp 8084/tcp 8084/udp 8085/tcp 8085/udp 8086/tcp 8086/udp 8087/tcp 8087/udp 8088/tcp 8088/udp 8089/tcp 8089/udp 5433/tcp 5433/udp 6370/tcp 6370/udp 8091/tcp 8091/udp 8092/tcp 8092/udp 8093/tcp 8093/udp 8094/tcp 8094/udp 8095/tcp 8095/udp 8096/tcp 8096/udp 8282/tcp 8282/udp 9988/tcp 9988/udp 3001/tcp 3001/udp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


ubuntu@my-vm:~$

Output of checks

This bug happens on my own server. Everything seems fine. I don't know what I could fix. The rest of the GitLab server is working. (Except for the Web IDE, which also shows blank files, and the GitLab console commands hang [For example: sudo gitlab-rails console takes 2-3 minutes to show the console])

Results of GitLab environment info

Expand for output related to GitLab environment info 
(I've replaced the actual URLs with https://git.mydomain.com - I don't want to expose the server.)
System information
System:         Ubuntu 20.04
Current User:   git
Using RVM:      no
Ruby Version:   2.7.4p191
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.16
Git Version:    2.33.1.
Sidekiq Version:6.2.2
Go Version:     unknown

GitLab information
Version:        14.5.1
Revision:       9d9ee598bc5
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://git.mydomain.com
HTTP Clone URL: https://git.mydomain.com/some-group/some-project.git
SSH Clone URL:  git@git.mydomain.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: auth0

GitLab Shell
Version:        13.22.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
3/2 ... yes
3/3 ... yes
4/4 ... yes
4/5 ... yes
8/7 ... yes
4/8 ... yes
4/9 ... yes
3/10 ... yes
3/11 ... yes
3/12 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 5
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixesgitlab.rb

None that I know of.

Edited by RedstoneWizard08