If for some reason gitlab-git-http-server is not properly configured, or you are using a custom nginx or Apache proxy that forwards directly to Unicorn and you attempt to clone via HTTP, the request succeeds, you receive a 200 and an empty repo.
This very much affects CI.
gitlab-ci-multi-runner 0.5.5-1-g69bc934 (69bc934) Using Shell executor... Running on ciworker01...Cloning repository... Cloning into 'builds/996c2f68/0/nms/project'... warning: You appear to have cloned an empty repository. Checking out 1b50f515 as dev... fatal: reference is not a tree: 1b50f5157b357f4c006ceccf0e36085bad300180ERROR: Build failed with: exit status 128
When this happens, the error should be more descriptive. Maybe mention that gitlab-git-http-server seems to not be running or something like that. It should also not respond with a 200.
@PaulBGD what problem are you experiencing? This issue is to show a better error message and not send an empty repo. Unicorn will no longer send repos over HTTP by default. This is now the job of the gitlab-git-http-server.
If you are getting the same warning: You appear to have cloned an empty repository. message, you should configure gitlab-git-http-server accordingly or disable it with gitlab_git_http_server['enable'] = false in your gitlab.rb file.
The cause of empty clones is Git clients talking to Unicorn when they should be talking to gitlab-git-http-server. Unicorn answers with data that gitlab-git-http-server understands (and needs), but Git clients do not. Because the response does not make sense the Git client decides to initialize a new repository for you.
@patricio people should not disable gitlab-git-http-server. That guarantees 'empty clones'.
I've had this problem and solved it by setting nginx['enabled'] = false in gitlab.rb. Then setting up apache2 using the file found here making sure the address and the port are all set to my set-up. Afterwards I changed gitlab.rb to set gitlab_git_http_server['listen_network'] = "tcp" and gitlab_git_http_server['listen_addr'] = "localhost:PORT" making sure that the port is the same as in the above apache2 VirtualHost file.
if people should not disable the gitlab-git-http-server why is there a setting that allows that?
@patricio all daemons in omnibus-gitlab can be disabled. On the backend server for gitlab.com we use this, for example, to only run Postgres and Redis.
I also use the omnibus package with a custom NGINX config. Everything was working great until the upgrade to 8. Does anyone know what I need to change in my NGINx config to get everything working again?
My full custom nginx file for omnibus installation for anyone else with the same problem.
upstream gitlab { server unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket;}upstream gitlab-git-http-server { server localhost:8181;}## Normal HTTP hostserver { ## Either remove "default_server" from the listen line below, ## or delete the /etc/nginx/sites-enabled/default file. This will cause gitlab ## to be served if you visit any address that your server responds to, eg. ## the ip address of the server (http://x.x.x.x/)n 0.0.0.0:80 default_server; listen 0.0.0.0:80; listen [::]:80; server_name gitlab.example.com; ## Replace this with something like gitlab.example.com server_tokens off; ## Don't show the nginx version number, a security best practice root /opt/gitlab/embedded/service/gitlab-rails/public; ## Increase this if you want to upload large attachments ## Or if you want to accept large git objects over http client_max_body_size 20m; ## See app/controllers/application_controller.rb for headers set ## Individual nginx logs for this GitLab vhost access_log /var/log/nginx/gitlab_access.log; error_log /var/log/nginx/gitlab_error.log; location / { ## Serve static files from defined root folder. ## @gitlab is a named location for the upstream fallback, see below. try_files $uri $uri/index.html $uri.html @gitlab; } ## We route uploads through GitLab to prevent XSS and enforce access control. location /uploads/ { ## If you use HTTPS make sure you disable gzip compression ## to be safe against BREACH attack. # gzip off; ## https://github.com/gitlabhq/gitlabhq/issues/694 ## Some requests take more than 30 seconds. proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Frame-Options SAMEORIGIN; proxy_pass http://gitlab; } ## If a file, which is not found in the root folder is requested, ## then the proxy passes the request to the upsteam (gitlab unicorn). location @gitlab { ## If you use HTTPS make sure you disable gzip compression ## to be safe against BREACH attack. # gzip off; ## https://github.com/gitlabhq/gitlabhq/issues/694 ## Some requests take more than 30 seconds. proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Frame-Options SAMEORIGIN; proxy_pass http://gitlab; } location ~ [-\/\w\.]+\.git\/ { ## If you use HTTPS make sure you disable gzip compression ## to be safe against BREACH attack. # gzip off; ## https://github.com/gitlabhq/gitlabhq/issues/694 ## Some requests take more than 30 seconds. proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; # Do not buffer Git HTTP responses proxy_buffering off; # The following settings only work with NGINX 1.7.11 or newer # # # Pass chunked request bodies to gitlab-git-http-server as-is # proxy_request_buffering off; # proxy_http_version 1.1; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://gitlab-git-http-server; } ## Enable gzip compression as per rails guide: ## http://guides.rubyonrails.org/asset_pipeline.html#gzip-compression ## WARNING: If you are using relative urls remove the block below ## See config/application.rb under "Relative url support" for the list of ## other files that need to be changed for relative url support location ~ ^/(assets)/ { root /opt/gitlab/embedded/service/gitlab-rails/public; gzip_static on; # to serve pre-gzipped version expires max; add_header Cache-Control public; } error_page 502 /502.html;}
I'm having the same issue with Apache and Gitlab installed from source. Not even sure I have a gitlab.rb file (is that an omnibus thing?)
gitlab-ci-multi-runner 0.5.5-1-g69bc934 (69bc934)Using SSH executor...stdin: is not a ttyRunning on gitlab.domain.com via gitlabrunner1...Cloning repository...Cloning into 'builds/ddb28055/0/default/repo-name'...warning: You appear to have cloned an empty repository.Checking out 02f2cef0 as master...fatal: reference is not a tree: 02f2cef0...80075aefERROR: Build failed with: Process exited with: 128. Reason was: ()
<VirtualHost *:80> ServerName gitlab.domain.com ServerSignature Off ProxyPreserveHost On AllowEncodedSlashes NoDecode <Location /> #Require all granted Order allow,deny Allow from all #Allow forwading to gitlab-git-http-server ProxyPassReverse http://127.0.0.1:8181 #Allow forwading to GitLab Rails app (Unicorn) ProxyPassReverse http://127.0.0.1:8080 ProxyPassReverse http://gitlab.domain.com/ </Location> RewriteEngine on #Forwad request ending with .git to gitlab-git-http-server RewriteCond %{REQUEST_URI} .*\.(git)$ RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA] RewriteCond %{REQUEST_URI} .*\.(git)/$ RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA] #Forwad forward any other requests to GitLab Rails app (Unicorn) RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA] # needed for downloading attachments DocumentRoot /home/git/gitlab/public #Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up. ErrorDocument 404 /404.html ErrorDocument 422 /422.html ErrorDocument 500 /500.html ErrorDocument 503 /deploy.html LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded ErrorLog /var/log/apache2/gitlab.domain.com_error.log CustomLog /var/log/apache2/gitlab.domain.com_forwarded.log common_forwarded CustomLog /var/log/apache2/gitlab.domain.com_access.log combined env=!dontlog CustomLog /var/log/apache2/gitlab.domain.com.log combined</VirtualHost>
Notice the rewrite condition for .git checks for end of line and possible trailing slash (otherwise files .gitignore redirect to the 503/deploy error page.
Should I bite the bullet and switch to nginx? Is there something else I can do with apache to make it work?
Git HTTP requests never end in .git or .git/; I think those $s don't belong there. A Git HTTP request looks like /namespace/repo.git/info/refs?service=git-upload-pack or /namespace/repo.git/git-upload-pack. So never ending in .git.
Maybe all this regex business is too complicated. Look at gitlab-git-http-server itself: it matches requests on the last part of the request URI, and there are really only three suffixes at the moment: /info/refs, /git-upload-pack and /git-receive-pack.
@jacobvosmaer Maybe what I'll try is matching .*\.(git)/.* and seeing if that works. Not sure what unintended consequences it'll have, but nothing ventured, nothing gained, right? (snapshots the vm)
@acatzr800 I think it is less risky/more robust to match the three strings I mentioned. The regex you propose can still catch requests file browser requests for files with interesting names.
I went from empty repo errors to
fatal: unable to access 'http://gitlab-ci-token:28..94@gitlab.domain.com/default/repo.git/': The requested URL returned error: 503
and Apache is yelling at me:
[Fri Sep 25 12:45:02 2015] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8181 (*) failed
So it appears that the redirect is working, but the connection is being denied
Hey there, for Apache please consult this MR gitlab-recipes!35 (merged) we are in the process of merging it.
Also, for source installations you have to edit /etc/default/gitlab as per this comment since apache is configured to listen on tcp and not on unix sockets.
@acatzr800 you need to edit /etc/default/gitlab and tell gitlab-git-http-server to listen on 127.0.0.1:8181; right now it is listening on a socket: -listenNetwork unix -listenAddr /home/git/gitlab/tmp/sockets/gitlab-git-http-server.socket
I've tried everything on this post and it still does not work. It seems that for some reason nginx is not following through the proxy_pass to that url. Any thoughts?
Configured gitlab-git-http-server to run on tcp localhost:8181
gitlab-git-http-server is running and when i run clone directly on the server on the 8181 port login is prompted.
We don't have the ominibus install because i'ved looked for that gitlab.rb file and is not available. Moreover, i have searched through the running programs and there is no other instance of gitlab-git-http-server.
However, if try to access remotely through http i get the empty repository warning.
➜ ~ git clone http://git.gbhapps.com/undefined/dominicana-solidaria.git
Cloning into 'dominicana-solidaria'...
warning: You appear to have cloned an empty repository.
Checking connectivity... done.
upstream gitlab {
server unix:/home/git/gitlab/tmp/sockets/gitlab.socket fail_timeout=0;
}
upstream gitlab-git-http-server {
# server unix:/home/git/gitlab/tmp/sockets/gitlab-git-http-server.socket fail_timeout=0;
server localhost:8181;
}
## Normal HTTP host
server {
## Either remove "default_server" from the listen line below,
## or delete the /etc/nginx/sites-enabled/default file. This will cause gitlab
## to be served if you visit any address that your server responds to, eg.
## the ip address of the server (http://x.x.x.x/)n 0.0.0.0:80 default_server;
listen 0.0.0.0:80 default_server;
listen [::]:80 default_server;
server_name git.gbhapp.com; ## Replace this with something like gitlab.example.com
server_tokens off; ## Don't show the nginx version number, a security best practice
root /home/git/gitlab/public;
## Increase this if you want to upload large attachments
## Or if you want to accept large git objects over http
client_max_body_size 20m;
## See app/controllers/application_controller.rb for headers set
## Individual nginx logs for this GitLab vhost
access_log /var/log/nginx/gitlab_access.log;
error_log /var/log/nginx/gitlab_error.log;
location / {
## Serve static files from defined root folder.
## @gitlab is a named location for the upstream fallback, see below.
try_files $uri $uri/index.html $uri.html @gitlab;
}
## We route uploads through GitLab to prevent XSS and enforce access control.
location /uploads/ {
## If you use HTTPS make sure you disable gzip compression
## to be safe against BREACH attack.
# gzip off;
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_pass http://gitlab;
}
## If a file, which is not found in the root folder is requested,
## then the proxy passes the request to the upsteam (gitlab unicorn).
location @gitlab {
## If you use HTTPS make sure you disable gzip compression
## to be safe against BREACH attack.
# gzip off;
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_pass http://gitlab;
}
location ~ [-\/\w\.]+\.git\/ {
## If you use HTTPS make sure you disable gzip compression
## to be safe against BREACH attack.
# gzip off;
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_redirect off;
# Do not buffer Git HTTP responses
proxy_buffering off;
# The following settings only work with NGINX 1.7.11 or newer
#
# # Pass chunked request bodies to gitlab-git-http-server as-is
# proxy_request_buffering off;
# proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://gitlab-git-http-server;
}
I have trouble with a local nginx/passenger installation.
I use the omnipackage, and the configuration here : https://gitlab.com/wakin/omnibus-gitlab/blob/master/doc/settings/nginx.md (Using a non-bundled web-server and Vhost (server block) for nginx)
No unix sockets are enabled by default without unicorn, and using 8181 redirection, the upstream is giving 500 error.
Thank you.
Edit : I found the problem.
Here's my conf :
gitlab.rb
upstream gitlab-git-http-server { server unix://var/opt/gitlab/gitlab-git-http-server/socket fail_timeout=0;}server { listen 80; listen 1999 ssl spdy; server_name fdn; server_tokens off; root /opt/gitlab/embedded/service/gitlab-rails/public; client_max_body_size 250m; access_log /var/log/nginx/gitlab_access.log; error_log /var/log/nginx/gitlab_error.log; # Ensure Passenger uses the bundled Ruby version passenger_ruby /opt/gitlab/embedded/bin/ruby; # Correct the $PATH variable to included packaged executables passenger_env_var PATH "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/local/bin:/usr/bin:/bin"; # Make sure Passenger runs as the correct user and group to # prevent permission issues passenger_user www-data; passenger_group www-data; # Enable Passenger & keep at least one instance running at all times passenger_enabled on; passenger_min_instances 1; location ~ [-\/\w\.]+\.git\/ { ## If you use HTTPS make sure you disable gzip compression ## to be safe against BREACH attack. gzip off; ## https://github.com/gitlabhq/gitlabhq/issues/694 ## Some requests take more than 30 seconds. proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; # Do not buffer Git HTTP responses proxy_buffering off; # The following settings only work with NGINX 1.7.11 or newer # # Pass chunked request bodies to gitlab-git-http-server as-is # proxy_request_buffering off; # proxy_http_version 1.1; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Ssl on; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://gitlab-git-http-server; }error_page 502 /502.html;
To resume :
The socket didn't worked because i had disabled unicorn before. So i enabled it. BUT it still didn't work... I find out few hours later, the permissions on unicorn stderr log file was not good and it didn't start !!
So my advice if you're in the same situation : check /var/log/gitlab/unicorn/current to see errors...
location ~ [-\/\w\.]+\.git\/ { ## If you use HTTPS make sure you disable gzip compression ## to be safe against BREACH attack. # gzip off; ## https://github.com/gitlabhq/gitlabhq/issues/694 ## Some requests take more than 30 seconds. proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; # Do not buffer Git HTTP responses proxy_buffering off; # The following settings only work with NGINX 1.7.11 or newer # # # Pass chunked request bodies to gitlab-git-http-server as-is # proxy_request_buffering off; # proxy_http_version 1.1; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://gitlab-git-http-server; }
You should really be using apt-listchanges to notify users of upgrades that require significant configuration changes. Clone-over-http isn't highly used in my setup so an outwardly-OK upgrade required a fair amount of troubleshooting to resolve.