Improve apache2 support and debug things

Hello,

I currently use the package provide by debian. I want to use it with apache2.

I have some issues and things i do not understand well.

The default recipe for apache seens not working as it.

So deep dive in my stuff :

• The gitlab-workhorse default port in recipe (8181) need to be replace by 8080. If not it lead to a 503 error.
• When that modification is done i can use some part of the web ui but for ie the api v4 and web edi not works. Error 404 for one and 422 for the other.
• When UI "works" i can not do git clone (http or ssh)

ssh : 

Cloning into 'public-test-project'...
Failed to read config, exiting
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists

http :

fatal: unable to access 'http://forge.intern.lan/public-test-group/public-test-project.git/': The requested URL returned error: 500

in the /var/log/apache2/gitlab/error.log

GET /public-test-group/public-test-project.git/info/refs?service=git-upload-pack HTTP/1.1" 500 2926

Track in /var/log/gitlab/production.log

Started GET "/public-test-group/public-test-project.git/info/refs?service=git-upload-pack" for 127.0.0.1 at 2021-06-02 03:19:04 +0200
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 86)
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 86)
Processing by Repositories::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "repository_path"=>"public-test-group/public-test-project.git"}
Completed 500 Internal Server Error in 29ms (ActiveRecord: 9.7ms | Elasticsearch: 0.0ms | Allocations: 4675)
  
JWT::DecodeError (Nil JSON web token):
  
lib/gitlab/jwt_authenticatable.rb:17:in `decode_jwt_for_issuer'
lib/gitlab/workhorse.rb:196:in `decode_jwt'
lib/gitlab/workhorse.rb:192:in `verify_api_request!'
app/controllers/concerns/workhorse_request.rb:13:in `verify_workhorse_api!'
app/controllers/application_controller.rb:485:in `set_current_admin'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:470:in `set_locale'
app/controllers/application_controller.rb:463:in `block in set_current_context'
lib/gitlab/application_context.rb:70:in `block in use'
lib/gitlab/application_context.rb:70:in `use'
lib/gitlab/application_context.rb:27:in `with_context'
app/controllers/application_controller.rb:454:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Env information :

LC_ALC_ALL=C gitlab-rake gitlab:env:info SANITIZE=true --trace

Check if Gitlab is configured correctly...
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Attention: used pure ruby version of MurmurHash3
DEPRECATION WARNING: ActionDispatch::Http::ParameterFilter is deprecated and will be removed from Rails 6.1. Use ActiveSupport::ParameterFilter instead. (called from <top (required)> at /usr/lib/ruby/vendor_ruby/grape_logging/util/parameter_filter.rb:2)
/usr/share/gitlab/lib/gitlab.rb:42: warning: already initialized constant Gitlab::COM_URL
/usr/share/gitlab/lib/gitlab.rb:42: warning: previous definition of COM_URL was here
/usr/share/gitlab/lib/gitlab.rb:43: warning: already initialized constant Gitlab::STAGING_COM_URL
/usr/share/gitlab/lib/gitlab.rb:43: warning: previous definition of STAGING_COM_URL was here
/usr/share/gitlab/lib/gitlab.rb:44: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
/usr/share/gitlab/lib/gitlab.rb:44: warning: previous definition of APP_DIRS_PATTERN was here
/usr/share/gitlab/lib/gitlab.rb:45: warning: already initialized constant Gitlab::SUBDOMAIN_REGEX
/usr/share/gitlab/lib/gitlab.rb:45: warning: previous definition of SUBDOMAIN_REGEX was here
/usr/share/gitlab/lib/gitlab.rb:46: warning: already initialized constant Gitlab::VERSION
/usr/share/gitlab/lib/gitlab.rb:46: warning: previous definition of VERSION was here
/usr/share/gitlab/lib/gitlab.rb:47: warning: already initialized constant Gitlab::INSTALLATION_TYPE
/usr/share/gitlab/lib/gitlab.rb:47: warning: previous definition of INSTALLATION_TYPE was here
/usr/share/gitlab/lib/gitlab.rb:48: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
/usr/share/gitlab/lib/gitlab.rb:48: warning: previous definition of HTTP_PROXY_ENV_VARS was here

System information
System:         Debian 10
Current User:   gitlab
Using RVM:      no
Ruby Version:   2.7.3p183
Gem Version:    3.1.6
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Bundler Version:2.1.4
Rake Version:   12.3.1
Redis Version:  6.0.12
Git Version:    2.31.1
Sidekiq Version:6.0.4
Go Version:     unknown

GitLab information
Version:        13.11.2
Revision:       Unknown
Directory:      /usr/share/gitlab
DB Adapter:     PostgreSQL
DB Version:     11.12
URL:            http://forge.intern.lan
HTTP Clone URL: http://forge.intern.lan/some-group/some-project.git
SSH Clone URL:  gitlab@intern.lan:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        13.17.0
Repository storage paths:
- default:      /var/lib/gitlab/repositories
GitLab Shell path:              /usr/share/gitlab-shell
Git:            /usr/bin/git

And the check (truncate) with things with error :

LC_ALL=C gitlab-rake gitlab:check SANITIZE=true --trace

Check if Gitlab is configured correctly...
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Attention: used pure ruby version of MurmurHash3
DEPRECATION WARNING: ActionDispatch::Http::ParameterFilter is deprecated and will be removed from Rails 6.1. Use ActiveSupport::ParameterFilter instead. (called from <top (required)> at /usr/lib/ruby/vendor_ruby/grape_logging/util/parameter_filter.rb:2)
/usr/share/gitlab/lib/gitlab.rb:42: warning: already initialized constant Gitlab::COM_URL
/usr/share/gitlab/lib/gitlab.rb:42: warning: previous definition of COM_URL was here
/usr/share/gitlab/lib/gitlab.rb:43: warning: already initialized constant Gitlab::STAGING_COM_URL
/usr/share/gitlab/lib/gitlab.rb:43: warning: previous definition of STAGING_COM_URL was here
/usr/share/gitlab/lib/gitlab.rb:44: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
/usr/share/gitlab/lib/gitlab.rb:44: warning: previous definition of APP_DIRS_PATTERN was here
/usr/share/gitlab/lib/gitlab.rb:45: warning: already initialized constant Gitlab::SUBDOMAIN_REGEX
/usr/share/gitlab/lib/gitlab.rb:45: warning: previous definition of SUBDOMAIN_REGEX was here
/usr/share/gitlab/lib/gitlab.rb:46: warning: already initialized constant Gitlab::VERSION
/usr/share/gitlab/lib/gitlab.rb:46: warning: previous definition of VERSION was here
/usr/share/gitlab/lib/gitlab.rb:47: warning: already initialized constant Gitlab::INSTALLATION_TYPE
/usr/share/gitlab/lib/gitlab.rb:47: warning: previous definition of INSTALLATION_TYPE was here
/usr/share/gitlab/lib/gitlab.rb:48: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
/usr/share/gitlab/lib/gitlab.rb:48: warning: previous definition of HTTP_PROXY_ENV_VARS was here

[...]

Running /usr/share/gitlab-shell/bin/check
Failed to read config, exiting
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u gitlab -H editor /usr/share/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

[...]

I already try several things and i m really confuse at this point. Mainly because I'm not an expert I guess.

This is my configuration files : the virtualhost Some link to help :

• https://stackoverflow.com/questions/4390436/need-to-allow-encoded-slashes-on-apache
• gitlab-foss#23133 (closed)

# This configuration has been tested on GitLab 13.6
# Note this config assumes unicorn/puma is listening on default port 8080 and
# gitlab-workhorse is listening on port 8181. 
# To make puma listen on port 8080 edit gitlab/config/puma.rb and add the following: 
#
# bind 'tcp://127.0.0.1:8080'
# 
# To allow gitlab-workhorse to listen on port 8181, edit or create 
# /etc/default/gitlab and change or add the following:
#
# gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr 127.0.0.1:8181 -authBackend http://127.0.0.1:8080"
#
#Module dependencies
# mod_rewrite
# mod_proxy
# mod_proxy_http
<VirtualHost *:80>

  ServerName forge.intern.lan
  ServerSignature Off

  ProxyPreserveHost On

  # Ensure that encoded slashes are not decoded but left in their encoded state.
  # http://doc.gitlab.com/ce/api/projects.html#get-single-project
  AllowEncodedSlashes NoDecode

  <Location />
    # New authorization commands for apache 2.4 and up
    # http://httpd.apache.org/docs/2.4/upgrading.html#access
    Require all granted

    #Allow forwarding to gitlab-workhorse
    ProxyPassReverse http://127.0.0.1:8080
    ProxyPassReverse http://forge.intern.lan/
  </Location>

 # Apache equivalent of nginx try files
  # http://serverfault.com/questions/290784/what-is-apaches-equivalent-of-nginxs-try-files
  # http://stackoverflow.com/questions/10954516/apache2-proxypass-for-rails-app-gitlab
  RewriteEngine on

  #Forward all requests to gitlab-workhorse except existing files like error documents
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
  RewriteCond %{REQUEST_URI} ^/uploads/.*
  RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA,NE]

  # needed for downloading attachments
  DocumentRoot /var/lib/gitlab/public

  #Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
  ErrorDocument 404 /404.html
  ErrorDocument 422 /422.html
  ErrorDocument 500 /500.html
  ErrorDocument 502 /502.html
  ErrorDocument 503 /503.html

  # It is assumed that the log directory is in /var/log/httpd.
  # For Debian distributions you might want to change this to
  # /var/log/apache2.
  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
  ErrorLog  /var/log/apache2/gitlab/error.log
  CustomLog /var/log/apache2/gitlab/forwarded.log common_forwarded
  CustomLog /var/log/apache2/gitlab/access.log combined env=!dontlog
  CustomLog /var/log/apache2/gitlab/gitlab.log combined

</VirtualHost>

Notes on file :

• On debian log can be write using ${APACHE_LOG_DIR}
• Test should be done on load modules with for ie

# https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypreservehost to get module provider.
<IfModule mod_proxy> 
 ProxyPreserveHost On
</IfModule>

On the debian package options can be modify on /default/gitlab

#! /bin/sh

# Copy this lib/support/init.d/gitlab.default.example file to
# /etc/default/gitlab in order for it to apply to your system.

# RAILS_ENV defines the type of installation that is running.
# Normal values are "production", "test" and "development".
RAILS_ENV="production"

# Read debian specific configuration
. /etc/gitlab/gitlab-debian.conf
. /etc/gitlab-common/gitlab-common.conf

for i in $(grep -v '#' /etc/gitlab/gitlab-debian.conf | cut -d= -f 1)
do
  export $i
done

# app_user defines the user that GitLab is run as.
# The default is "git".
export app_user=${gitlab_user}

# app_root defines the folder in which gitlab and it's components are installed.
# The default is "/home/$app_user/gitlab"
app_root=${gitlab_app_root}

# gitlab_log_dir is defined in /etc/gitlab/gitlab-debian.conf

# pid_path defines a folder in which the gitlab and it's components place their pids.
# This variable is also used below to define the relevant pids for the gitlab components.
# The default is "$app_root/tmp/pids"
pid_path="${gitlab_pid_path}"

# socket_path defines the folder in which gitlab places the sockets
#The default is "$app_root/tmp/sockets"
socket_path="${gitlab_pid_path}"

# web_server_pid_path defines the path in which to create the pid file fo the web_server
# The default is "$pid_path/unicorn.pid"
web_server_pid_path="$pid_path/unicorn.pid"
# sidekiq_logfile defines log file used by sidekiq
sidekiq_logfile="${gitlab_log_dir}/sidekiq.log"

gitlab_workhorse_pid_path="$pid_path/gitlab-workhorse.pid"
# The -listenXxx settings determine where gitlab-workhorse
# listens for connections from NGINX. To listen on localhost:8181, write
# '-listenNetwork tcp -listenAddr localhost:8181'.
# The -authBackend setting tells gitlab-workhorse where it can reach
# Unicorn.
#gitlab_workhorse_options="-listenUmask 0 -listenNetwork unix -listenAddr $socket_path/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080"
#gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr 127.0.0.1:8181 -authBackend http://127.0.0.1:8080"
#gitlab_workhorse_options="-listenNetwork tcp -listenAddr 127.0.0.1:8181 -authBackend http://127.0.0.1:8080"
#gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr 127.0.0.1:8181 -authBackend http://127.0.0.1:8080" 
gitlab_workhorse_options="-listenUmask 0 -listenNetwork unix -listenAddr /run/gitlab/gitlab-workhorse.socket -authSocket /run/gitlab/gitlab.socket -documentRoot /var/lib/gitlab/public"
gitlab_workhorse_log="${gitlab_log_dir}/gitlab-workhorse.log"

# mail_room_enabled specifies whether mail_room, which is used to process incoming email, is enabled.
# This is required for the Reply by email feature.
# The default is "false"
mail_room_enabled=false

# mail_room_pid_path defines the path in which to create the pid file for mail_room
# The default is "$pid_path/mail_room.pid"
mail_room_pid_path="$pid_path/mail_room.pid"

# mail_room_logfile defines log file used by mailroom
mail_room_logfile="${gitlab_log_dir}/mail_room.log"

# shell_path defines the path of shell for "$app_user" in case you are using
# shell other than "bash"
# The default is "/bin/bash"
shell_path="/bin/bash"

And in the /etc/gitlab/*

• gitlab.yml -> nothing special here (i can provide if needed)
• puma.rb

# frozen_string_literal: true

# Load "path" as a rackup file.
#
# The default is "config.ru".
#
rackup 'config.ru'
pidfile "#{ENV['gitlab_pid_path']}/puma.pid"
state_path "#{ENV['gitlab_pid_path']}/puma.state"
# frozen_string_literal: true

# Load "path" as a rackup file.
#
# The default is "config.ru".
#
rackup 'config.ru'
pidfile "#{ENV['gitlab_pid_path']}/puma.pid"
state_path "#{ENV['gitlab_pid_path']}/puma.state"

stdout_redirect File.join(ENV['gitlab_log_dir'],"puma.stdout.log"),
  File.join(ENV['gitlab_log_dir'],"puma.stderr.log"),
  true

# Configure "min" to be the minimum number of threads to use to answer
# requests and "max" the maximum.
#
# The default is "0, 16".
#
threads 1, 16

# By default, workers accept all requests and queue them to pass to handlers.

stdout_redirect File.join(ENV['gitlab_log_dir'],"puma.stdout.log"),
  File.join(ENV['gitlab_log_dir'],"puma.stderr.log"),
  true

# Configure "min" to be the minimum number of threads to use to answer
# requests and "max" the maximum.
#
# The default is "0, 16".
#
threads 1, 16

# By default, workers accept all requests and queue them to pass to handlers.

# When false, workers accept the number of simultaneous requests configured.
#
# Queueing requests generally improves performance, but can cause deadlocks if
# the app is waiting on a request to itself. See https://github.com/puma/puma/issues/612
#
# When set to false this may require a reverse proxy to handle slow clients and
# queue requests before they reach puma. This is due to disabling HTTP keepalive
queue_requests false

# Bind the server to "url". "tcp://", "unix://" and "ssl://" are the only
# accepted protocols.
#bind "unix://#{ENV['gitlab_pid_path']}/gitlab.socket"
# bind 'tcp://127.0.0.1:8080'
bind 'tcp://0.0.0.0:8080'


workers 3

require_relative "#{ENV['gitlab_app_root']}/lib/gitlab/cluster/lifecycle_events"
require_relative "#{ENV['gitlab_app_root']}/lib/gitlab/cluster/puma_worker_killer_initializer"

on_restart do
  # Signal application hooks that we're about to restart
  Gitlab::Cluster::LifecycleEvents.do_before_master_restart
end


before_fork do
  # Signal to the puma killer
  Gitlab::Cluster::PumaWorkerKillerInitializer.start @config.options unless ENV['DISABLE_PUMA_WORKER_KILLER']

  # Signal application hooks that we're about to fork
  Gitlab::Cluster::LifecycleEvents.do_before_fork
end

Gitlab::Cluster::LifecycleEvents.set_puma_options @config.options
on_worker_boot do
  # Signal application hooks of worker start
  Gitlab::Cluster::LifecycleEvents.do_worker_start
end

# Preload the application before starting the workers; this conflicts with
# phased restart feature. (off by default)
preload_app!

tag 'gitlab-puma-worker'

# Verifies that all workers have checked in to the master process within
# the given timeout. If not the worker process will be restarted. Default
# value is 60 seconds.
#
worker_timeout 60

# Use json formatter
require_relative "#{ENV['gitlab_app_root']}/lib/gitlab/puma_logging/json_formatter"

json_formatter = Gitlab::PumaLogging::JSONFormatter.new
log_formatter do |str|
  json_formatter.call(str)
end