404 on CSS file
I just updated to 8.12 via the omnibus package, and now gitlab is returning a 404 loading /assets/application-1840f43fd20c30180b0599c214ddae7e6939d50933870904ebd0e81704658333.css, which is breaking the page layout.
The file exists:
# ls -l /opt/gitlab/embedded/service/gitlab-rails/public/assets/application-1840f43fd20c30180b0599c214ddae7e6939d50933870904ebd0e81704658333.css
-rw-r--r--. 1 root root 611471 Sep 22 11:52 /opt/gitlab/embedded/service/gitlab-rails/public/assets/application-1840f43fd20c30180b0599c214ddae7e6939d50933870904ebd0e81704658333.cssAnother CSS file in the same directory loads fine:
# ls -l /opt/gitlab/embedded/service/gitlab-rails/public/assets/print-68eed6d8135d858318821e790e25da27b2b4b9b8dbb1993fa6765d8e2e3e16ee.css
-rw-r--r--. 1 root root 620 Sep 22 11:52 /opt/gitlab/embedded/service/gitlab-rails/public/assets/print-68eed6d8135d858318821e790e25da27b2b4b9b8dbb1993fa6765d8e2e3e16ee.cssHere is what production.log has to say:
Started GET "/assets/application-1840f43fd20c30180b0599c214ddae7e6939d50933870904ebd0e81704658333.css" for xxx.xxx.xxx.xxx at 2016-09-23 09:37:11 -0400
Processing by ProjectsController#show as HTML
Parameters: {"namespace_id"=>"assets", "id"=>"application-1840f43fd20c30180b0599c214ddae7e6939d50933870904ebd0e81704658333.css"}
Filter chain halted as :project rendered or redirected
Completed 404 Not Found in 22ms (Views: 0.5ms | ActiveRecord: 2.1ms)Please advise.
Activity
This seems similar to https://gitlab.com/gitlab-org/gitlab-ce/issues/14558, but I am not using a relative URL.
I just got the exact same issue. Upgrading omnibus installation from 8.11.7 to 8.12.0 with a reverse nginx proxy. Is a different file, but essentially the same problem...
==> /var/log/gitlab/gitlab-rails/production.log <== Processing by RootController#index as HTML Completed 200 OK in 58ms (Views: 27.6ms | ActiveRecord: 3.8ms) Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-09-23 15:42:54 +0000 Started POST "/ci/api/v1/builds/register.json" for xxx.xxx.xxx.xxx at 2016-09-23 15:42:54 +0000 Started GET "/assets/application-204e7b3d5eb4881844cf4ccd8b6a8a0d60ca35d2fda088c667058efbc4aa9499.css" for xxx.xxx.xxx.xxx at 2016-09-23 15:42:54 +0000 Processing by ProjectsController#show as HTML Parameters: {"namespace_id"=>"assets", "id"=>"application-204e7b3d5eb4881844cf4ccd8b6a8a0d60ca35d2fda088c667058efbc4aa9499.css"} Filter chain halted as :project rendered or redirected Completed 404 Not Found in 16ms (Views: 0.5ms | ActiveRecord: 1.6ms)Thank you very much for your help.
Edited by Esteban Nagy
Same issue here. I'm using Archlinux and I installed GitLab v8.11 successfully a couple of weeks ago following the official wiki. Yesterday I updated to v8.12 and now I'm facing the same problem.
Edited by Baptiste RebillardMentioned in issue #22011 (closed)
Can you give us the output of the check and info rake tasks?
Results of the the GitLab health check
- Installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true - For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
Results of GitLab environment info
- For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info - For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
Edited by Connor Shea- Installations with omnibus-gitlab package run and paste the output of:
-
@connorshea Here are mines:
[root@portable-b initializers]# cd /usr/share/webapps/gitlab [root@portable-b gitlab]# sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true rake aborted! LoadError: cannot load such file -- /usr/share/webapps/gitlab/config/application /usr/share/webapps/gitlab/Rakefile:5:in `require' /usr/share/webapps/gitlab/Rakefile:5:in `<top (required)>' /usr/bin/bundle:23:in `load' /usr/bin/bundle:23:in `<main>' (See full trace by running task with --trace) [root@portable-b gitlab]# sudo -u git -H bundle exec rake gitlab:env:info rake aborted! LoadError: cannot load such file -- /usr/share/webapps/gitlab/config/application /usr/share/webapps/gitlab/Rakefile:5:in `require' /usr/share/webapps/gitlab/Rakefile:5:in `<top (required)>' /usr/bin/bundle:23:in `load' /usr/bin/bundle:23:in `<main>' (See full trace by running task with --trace)However running these command lines following the official Archlinux wiki returns:
[root@portable-b gitlab]# cd /usr/share/webapps/gitlab [root@portable-b gitlab]# sudo -u gitlab bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true Checking GitLab Shell ... GitLab Shell version >= 3.6.0 ? ... OK (3.6.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by gitlab:gitlab? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/14 ... ok 5/15 ... ok 6/16 ... repository is empty 7/17 ... repository is empty 7/18 ... repository is empty 7/19 ... repository is empty 6/20 ... repository is empty 7/21 ... repository is empty Running /usr/share/webapps/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/lib/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Reply by email ... Reply by email is disabled in config/gitlab.yml Checking Reply by email ... Finished Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... no Try fixing it: sudo chmod 700 /var/lib/gitlab/uploads For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors projects have namespace: ... 2/14 ... yes 5/15 ... yes 6/16 ... yes 7/17 ... yes 7/18 ... yes 7/19 ... yes 6/20 ... yes 7/21 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.1) Your git bin path is "/usr/bin/git" Git version >= 2.7.3 ? ... yes (2.10.0) Active users: 4 Checking GitLab ... Finished [root@portable-b gitlab]# sudo -u gitlab bundle exec rake gitlab:env:info RAILS_ENV=production System information System: Current User: gitlab Using RVM: no Ruby Version: 2.3.1p112 Gem Version: 2.5.1 Bundler Version:1.13.1 Rake Version: 10.5.0 Sidekiq Version:4.1.4 GitLab information Version: 8.12.0 Revision: fatal: Ni ceci ni aucun de ses répertoires parents n'est un dépôt git : .git Directory: /usr/share/webapps/gitlab DB Adapter: mysql2 URL: http://cluxter.net:8088 HTTP Clone URL: http://cluxter.net:8088/some-group/some-project.git SSH Clone URL: gitlab@cluxter.net:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 3.6.0 Repository storage paths: - default: /var/lib/gitlab/repositories/ Hooks: /usr/share/webapps/gitlab-shell/hooks/ Git: /usr/bin/git -
@connorshea These are for my omnibus installation. Also considering in rolling back, specially for the weekend...
root <at> server$ gitlab-rake gitlab:env:info System information System: Debian 8.6 Current User: git Using RVM: no Ruby Version: 2.3.1p112 Gem Version: 2.6.6 Bundler Version:1.13.1 Rake Version: 10.5.0 Sidekiq Version:4.1.4 GitLab information Version: 8.12.0 Revision: 0864ff9 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: xxxxxxxxxxxxxxxxxxxxxxx HTTP Clone URL: xxxxxxxxxxxxxxxxx/some-group/some-project.git SSH Clone URL: git@xxxxxxxxxxxxxxxxxxxxxxxx:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 3.6.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/ Git: /opt/gitlab/embedded/bin/git root <at> server$ gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ... GitLab Shell version >= 3.6.0 ? ... OK (3.6.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 13/1 ... repository is empty 12/2 ... ok 19/3 ... ok 30/4 ... ok 13/5 ... repository is empty 32/7 ... ok 33/8 ... ok 34/9 ... ok 32/10 ... ok 34/11 ... ok 34/12 ... ok 11/13 ... repository is empty 34/14 ... ok 34/15 ... ok 19/17 ... ok 34/18 ... ok 34/19 ... ok 34/20 ... ok 27/23 ... ok 27/25 ... ok 34/26 ... repository is empty Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Reply by email ... Reply by email is disabled in config/gitlab.yml Checking Reply by email ... Finished Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ... 13/1 ... yes 12/2 ... yes 19/3 ... yes 30/4 ... yes 13/5 ... yes 32/7 ... yes 33/8 ... yes 34/9 ... yes 32/10 ... yes 34/11 ... yes 34/12 ... yes 11/13 ... yes 34/14 ... yes 34/15 ... yes 19/17 ... yes 34/18 ... yes 34/19 ... yes 34/20 ... yes 27/23 ... yes 27/25 ... yes 34/26 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.1) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.7.4) Active users: 37 Checking GitLab ... FinishedEdited by Esteban Nagy I've attached mine.
Milestone changed to %8.12
-
@brianodonnell that's fine, sorry you had to rollback :/
@cluxter @enagy @svenabels Thanks, we're looking into it now.
-
I fixed the Upload folder and ran these commands:
[root@portable-b gitlab]# sudo -u gitlab bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true Checking GitLab Shell ... GitLab Shell version >= 3.6.0 ? ... OK (3.6.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by gitlab:gitlab? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/14 ... ok 5/15 ... ok 6/16 ... repository is empty 7/17 ... repository is empty 7/18 ... repository is empty 7/19 ... repository is empty 6/20 ... repository is empty 7/21 ... repository is empty Running /usr/share/webapps/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/lib/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Reply by email ... Reply by email is disabled in config/gitlab.yml Checking Reply by email ... Finished Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... yes Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors projects have namespace: ... 2/14 ... yes 5/15 ... yes 6/16 ... yes 7/17 ... yes 7/18 ... yes 7/19 ... yes 6/20 ... yes 7/21 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.1) Your git bin path is "/usr/bin/git" Git version >= 2.7.3 ? ... yes (2.10.0) Active users: 4 Checking GitLab ... Finished [root@portable-b gitlab]# sudo -u gitlab -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production [root@portable-b gitlab]# systemctl restart gitlab-unicorn gitlab-sidekiq gitlab-workhorseStill no luck though :-/
In production, Rails should not be serving these static assets. The
production.logindicates it's incorrectly getting to the Rails controller for some reason. It should be going through gitlab-workhorse, and you should see some log message such as:2016-09-23_20:05:52.66931 worker9 gitlab-workhorse: 2016/09/23 20:05:52 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/assets/select2x2-6fe28d687dc0ed4d96016238c608ba1e7198c9c9accfa0b360b78018b9fb9bc2.png" ("") for GET "/assets/select2x2-6fe28d687dc0ed4d96016238c608ba1e7198c9c9accfa0b360b78018b9fb9bc2.png"Can you check your gitlab-workhorse logs to see if there are any errors?
Sure. The favicon and the print...css seem fine but the JS and the CSS show a 404 altough the files are in the asset folder. Here is the log tail:
Started GET "/" for 91.97.38.233 at 2016-09-23 22:19:12 +0200 Processing by RootController#index as HTML Completed 200 OK in 171ms (Views: 60.5ms | ActiveRecord: 21.2ms) Started GET "/assets/application-c8d3f635e65582c3f37efa5258a1edf1832f5d0b02e2fb30d8303a2ae60a5692.css" for 91.97.38.233 at 2016-09-23 22:19:12 +0200 Started GET "/assets/application-591156345de7cf30941cac4744c4744835cb4639743d7a48c884017bc0fc486c.js" for 91.97.38.233 at 2016-09-23 22:19:12 +0200 Processing by ProjectsController#show as HTML Parameters: {"namespace_id"=>"assets", "id"=>"application-c8d3f635e65582c3f37efa5258a1edf1832f5d0b02e2fb30d8303a2ae60a5692.css"} Filter chain halted as :project rendered or redirected Completed 404 Not Found in 38ms (Views: 1.0ms | ActiveRecord: 5.7ms) Started GET "/assets/print-68eed6d8135d858318821e790e25da27b2b4b9b8dbb1993fa6765d8e2e3e16ee.css" for 91.97.38.233 at 2016-09-23 22:19:12 +0200 Started GET "/assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico" for 91.97.38.233 at 2016-09-23 22:19:12 +0200(I did a rake cache:clear already several times and also an assets:precompile, a reconfigure and a restart)
-
People, we just found an issue that was open about a year ago, this one...
Seems that something went wrong with the special attributes for git user. In my case, we just run
setfacl -R -m u:git:rwX /opt/gitlab/embedded/service/gitlab-rails/public/assets/And then, gitlab-ctl reconfigure, gitlab-ctl restart
Works like a charm right now... so, I'm not upgrading automatically my server anymore. Every update worked fine until this one. My server is running an omnibus installation with Debian 8 (jessie), for reference.
Thank you very much people!!
@enagy I tried that but it didn't change anything. :-(
@brianodonnell @cluxter @svenabels is your reverse proxies pointing at gitlab's nginx, or gitlab's unicorn server, or the workhorse? They need to either be pointed at workhorse or nginx, and not the unicorn server.
-
@twk3 It was working before the upgrade so if it's not pointing to the right server anymore it means the upgrade changed it - which I guess is unlikely, right?
I'm not sure where to check that exactly but in
/etc/webapps/gitlab/gitlab.ymlmy settings didn't change vs v8.11. Under thegitlabsection I still havehost: cluxter.net,port: 8088andhttps: falsewhich is what I set for v8.11. Undertrusted_proxies:the 3 examples are still commented out. @twk3 For me, it's pointing to the unicorn port but that's always been the case in the last years and it always worked fine. nginx['enable'] is set to false in our case
@twk3 @cluxter I just changed the reverse proxy to point to nginx and I switched on nginx in gitlab and now it works fine. :-) However, I wonder why this is the case because pointing to unicorn worked like a charm for all the time so far - until upgrading to 8.12.0
-
@twk3 I'm still new with the GitLab configuration (I just installed it last week for the first time) so I'm not totally familiar with how the different elements of GitLab work together or what they even are. Could you please give us more details about this change? For example what file(s) did you change? Thanks!
Sure. I've set the following properties at /etc/gitlab/gitlab.rb:
external_url 'http://XXXXX:9292' gitlab_url 'http://XXXXX:9292/' gitlab_rails['internal_api_url'] = 'http://XXXXX:9292/'Also:
#nginx['enable'] = false-> This has been switched off before for us. I now commented this out.Our external_url previously pointed to Port 8888, which was from our unicorn setting:
unicorn['port'] = 8888-
@svenabels From what I saw this file is used by the omnibus package, right? I'm trying - hard - to make it work from the source package which is not very easy. The
gitlab-workhorseservice is installed and running according tosystemctl status gitlab-workhorsebut I didn't find how to configure the reverse proxy to point at workhorse yet, as suggested by @twk3 . I need to understand how the workhorse has been configured first, then how to configure the reverse proxy to make it point at the workhorse. Any help greatly appreciated from anyone who has experience in configuring a GitLab from the source ;-) -
I'm not sure what changed in this recent release yet relating to this, but we do intend for people to use the workhorse instead of of directly hitting unicorn. The workhorse handles static files, git over http, repository archiving etc.
If using apache, (or you want to and connect to workhorse on a port instead of the default unix socket), note the workhorse settings that need to change here: See the text under
gitlab-workhorse settings if using Apachehere: https://docs.gitlab.com/omnibus/settings/nginx.html#using-a-non-bundled-web-serverAnd here we have examples of what the reverse proxies should look like for nginx and apache for pointing at workhorse: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server
Edited by DJ Mountney -
@cluxter For a source install you may have to adjust workhorse's startup flags to make sure it's on a port, if your reverse proxy is apache or it's nginx but on a different machine: https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md
That doesn't solve the problem most of us are having. The issue is that ALL CSS and JS in the /assets/ folder work
ONLY the application-*.css isn't working
Edited by AlexEdimensionz-
@twk3 is right here--again, if you are seeing errors in the Rails log, that means that something is routing the
/assets/XXXURL incorrectly to unicorn.For those of you having trouble, it would really help if you include the settings of your reverse proxy--particularly those of you who are using your own instead of the one managed by omnibus. gitlab-workhorse logs may also help.
Edited by Stan Hu 
thanks @stanhu . for me, change nginx configuration file from
upstream gitlab-workhorse { server unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fail_timeout=0; } ......to
upstream gitlab-workhorse { server unix:/var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0; } ......is helpful.
i use centos 6 and install gitlab by yum package manager.
other configuration files can be found in https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server
for some guys who locates here by search engine,if you got 404 with all static files, DO NOT change you reverse proxy settings from
proxy_pass http://gitlab-workhorse;toproxy_pass https://gitlab-workhorse;sorry of my poor english.but maybe it's helpful to someone.
Edited by TLDay
Same problem here. After updating the Ubuntu omnibus package from 8.11.x to 8.12, CSS files result in a 404. Running behind an nginx reverse proxy, but the problem persists when trying to access the css files locally via curl (
curl --cookie '_gitlab_session=…' -v http://localhost:8080/assets/application-891a61baf08dd362204cccb62419682e810754e7b9e657eb3d33897e53d5bd96.css).Hello,
Same error for me 8.11 to 8.12 => 302 Not Found on application-*.css
sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ... GitLab Shell version >= 3.6.0 ? ... OK (3.6.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 3/1 ... repository is empty 5/2 ... ok 5/3 ... ok 5/8 ... ok 5/9 ... ok 5/10 ... ok 5/11 ... ok 5/12 ... ok 5/13 ... ok 5/14 ... ok 5/15 ... ok 5/16 ... ok 5/17 ... ok 5/18 ... ok 5/19 ... ok 5/20 ... ok 5/21 ... ok 5/22 ... ok 5/23 ... ok 5/24 ... ok 5/25 ... ok 5/26 ... ok 5/28 ... ok 5/29 ... ok 5/31 ... ok 5/32 ... ok 5/33 ... ok 5/34 ... ok 5/35 ... ok 5/36 ... ok 5/37 ... ok 5/38 ... ok 5/39 ... ok 5/40 ... ok 5/41 ... ok 5/42 ... ok 5/43 ... ok 5/44 ... ok 5/45 ... ok 5/46 ... ok 5/47 ... ok 5/48 ... ok 5/50 ... ok 5/51 ... ok 5/52 ... ok 5/53 ... ok 10/54 ... ok 11/55 ... repository is empty 6/56 ... ok 5/57 ... ok 5/59 ... ok 11/61 ... ok 5/62 ... ok 5/63 ... ok 5/64 ... ok 11/65 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Reply by email ... Reply by email is disabled in config/gitlab.yml Checking Reply by email ... Finished Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ... 3/1 ... yes 5/2 ... yes 5/3 ... yes 5/8 ... yes 5/9 ... yes 5/10 ... yes 5/11 ... yes 5/12 ... yes 5/13 ... yes 5/14 ... yes 5/15 ... yes 5/16 ... yes 5/17 ... yes 5/18 ... yes 5/19 ... yes 5/20 ... yes 5/21 ... yes 5/22 ... yes 5/23 ... yes 5/24 ... yes 5/25 ... yes 5/26 ... yes 5/28 ... yes 5/29 ... yes 5/31 ... yes 5/32 ... yes 5/33 ... yes 5/34 ... yes 5/35 ... yes 5/36 ... yes 5/37 ... yes 5/38 ... yes 5/39 ... yes 5/40 ... yes 5/41 ... yes 5/42 ... yes 5/43 ... yes 5/44 ... yes 5/45 ... yes 5/46 ... yes 5/47 ... yes 5/48 ... yes 5/50 ... yes 5/51 ... yes 5/52 ... yes 5/53 ... yes 10/54 ... yes 11/55 ... yes 6/56 ... yes 5/57 ... yes 5/59 ... yes 11/61 ... yes 5/62 ... yes 5/63 ... yes 5/64 ... yes 11/65 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.1) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.7.4) Active users: 8 Checking GitLab ... Finishedsudo gitlab-rake gitlab:env:info System information System: Debian 8.6 Current User: git Using RVM: no Ruby Version: 2.3.1p112 Gem Version: 2.6.6 Bundler Version:1.13.1 Rake Version: 10.5.0 Sidekiq Version:4.1.4 GitLab information Version: 8.12.1 Revision: ca3c6a4 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://gitlab.inwebo.com HTTP Clone URL: http://gitlab.company.com/some-group/some-project.git SSH Clone URL: git@gitlab.company.com:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 3.6.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/ Git: /opt/gitlab/embedded/bin/gitI'm also behind nginx reverse proxy
upstream gitlab-workhorse { server 127.0.0.1:8080 fail_timeout=0; }If I try
server unix:/var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0;its failed with 502.I fixed the error by following this topic: https://docs.gitlab.com/omnibus/settings/nginx.html#using-a-non-bundled-web-server
- In /etc/gitlab/gitlab.rb: nginx was already disabled
- In /etc/gitlab/gitlab.rb: external_users was not setted so I did it and executed "gitlab-ctl reconfigure"
- I didn't put trusted_proxies because nginx is not the same server
- I downloaded the last nginx file https://gitlab.com/gitlab-org/gitlab-recipes/blob/master/web-server/nginx/gitlab-omnibus-nginx.conf and replace the YOUR_SERVER_FQDN by gitlab.company.com
And it's work now !!!
@stanhu @connorshea is the Frontend label correct?
-
@twk3 Thank you so much for your advice about using gitlab-workhorse instead of gitlab-unicorn! I configured gitlab-workhorse correctly and I am now able to use it instead of gitlab-unicorn, which almost solved the problem (and it's much faster).
However my avatar is not showing up anymore now. It is successfully uploaded into the
/var/lib/gitlab/uploads/user/avatar/2/folder though (I am using Archlinux): the timestamp and size are correctly updated everytime I upload a new avatar and this is the right file because I was able to open it in an image viewer and see that it was my file. The permissions weren't set right at first according to the checking tool, so I followed the advice from the checking tool to modify permissions but no luck. I then set 777 on theuploadsfolder, it didn't work. I then also set 777 on its recursive folders, still no luck. I also tried this:https://gitlab.com/gitlab-org/gitlab-ce/issues/341
and consequently this:
https://github.com/gitlabhq/gitlabhq/issues/5430#issuecomment-36205393
but it didn't change anything (I obviously reloaded and restarted all the modules everytime I changed something).
I am not using nginx or Apache at all - only gitlab-workhorse.
I tried uploading other files as well in an issue that I have in my project. The file is successfully uploaded as
/var/lib/gitlab/uploads/cluxter/test1/bce78ce6d035702303ef795ed0c36acc/Test.odt, I can see it and use it. But I can't download it from my GitLab. The browser keeps waiting until it times out.If you (or anyone else) could give me another clue on how I could investigate this issue I would be very grateful.
PS: I will write a whole description of the steps I followed to solve all this once I will be done with the avatar issue.
EDIT: I fixed the avatar / uploaded files issue following this: https://serverfault.com/questions/752921/gitlab-and-apache-uploaded-images-404-errors
Edited by Baptiste Rebillard -
Here is how I fixed this issue for Archlinux. Big thanks to @twk3 who gave me the key advice to unlock it. Please do not hesitate to correct me if I'm wrong somewhere or to clarify some points.
Definitions
GitLab can rely on several parts:
-
gitlab-unicorn: this is an HTTP server which can also act as a reverse HTTP proxy -
gitlab-workhorse: from what I understood, basically the same thing thangitlab-unicornbut much faster, more modern and better suited for big HTTP requests. Sogitlab-workhorseshould be preferred overgitlab-unicorn. The GitLab team intends for people to use it instead ofgitlab-unicorn -
apache: also an HTTP server + reverse HTTP proxy, so basically the same role than the 2 previous packages -
nginx: likeapachebut usually with better performances and an easier configuration
My situation
I use Archlinux and I didn't want to use
apachenornginx. So far my GitLab was relying ongitlab-unicorn. I understood I should usegitlab-workhorseinstead since some unknown changes happened ongitlab-unicornthat made it unusable. I knew I had installedgitlab-workhorsebut I didn't know how it was set up.The big change
First I had to see if
gitlab-workhorsewas loaded:# systemctl status gitlab-workhorse ● gitlab-workhorse.service - Gitlab Workhorse Loaded: loaded (/usr/lib/systemd/system/gitlab-workhorse.service; disabled; vendor preset: disabled) Active: active (running) since lun. 2016-09-26 15:47:18 CEST; 29min ago Process: 7833 ExecStop=/usr/bin/kill -QUIT $MAINPID (code=exited, status=0/SUCCESS) Main PID: 7847 (gitlab-workhors) Tasks: 6 (limit: 4915) CGroup: /system.slice/gitlab-workhorse.service └─7847 /usr/bin/gitlab-workhorse -listenUmask 0 -authBackend http://localhost:8080 -authSocket /var/lib/gitlab/sockets/gitlab.socket -listenAddr /run/gitlab/gitlab-workhorse.socket -listenNetwork unix /var/lib/gitlab/repositories -documentRoot /usr/share/webapps/gitlab/public/usr/share/webapps/gitlab/publicIt was loaded and running but I saw the
listenAddrparameter was set up on a UNIX socket. I used this manual to understand this: https://gitlab.com/gitlab-org/gitlab-workhorseSo I understood I had to change this parameter to use a TCP connexion on a network interface instead of using a socket. Thus I edited the service config file like this (path given by the "systemctl status" command):
# vim /usr/lib/systemd/system/gitlab-workhorse.service _____ BOF _____ [Service] User=gitlab Group=gitlab WorkingDirectory=/usr/share/webapps/gitlab SyslogIdentifier=gitlab-workhorse CapabilityBoundingSet= ProtectSystem=full ProtectHome=true NoNewPrivileges=true ExecStart=/usr/bin/gitlab-workhorse -listenUmask 0 -authBackend http://localhost:8080 -authSocket /var/lib/gitlab/sockets/gitlab.socket -listenAddr 192.168.0.1:8181 -documentRoot /usr/share/webapps/gitlab/public ExecStop=/usr/bin/kill -QUIT $MAINPID ExecReload=/usr/bin/kill -USR2 $MAINPID Restart=on-failure RestartSec=1 [Install] WantedBy=multi-user.target _____ EOF _____To make it easier, the only thing I changed was this parameter on the
ExecStartligne:-listenAddr 192.168.0.1:8181I restarted the service:
# systemctl daemon-reload # systemctl restart gitlab-workhorseI was then able to access my GitLab through http://192.168.0.1:8181. Success!
-
-
unicorn is still the server used internally for serving up the dynamic app pages. But it has a limited number for workers for dealing with long lived requests, it isn't as fast at serving static assets as the servers most of us are familiar with, and it has request timeouts geared towards connections coming in from the browser. (unlike coming in over a git push/fetch for example).
So instead, workhorse is dealing with all those things that unicorn is bad at, and proxying requests to unicorn for the stuff that unicorn is good at. (serving the dynamic pages)
The nginx or apache in front is now mostly only used for ssl termination if you are using https, working nicer with other sites you may be hosting on the box, and providing a well-known exposed system for things like your firewall.
We are seeing the same problem in a HA deployment when 2 or more servers are running. It turns out that the
application-[hash].cssfile is not the same on every server. In fact, it changes when regenerating the assets. I attached a diff with the changes between asset compilation. It seems that keyframes and animation names are randomized. Therefore, the resulting hash is different causing 404 error.This seems to be the trouble maker: https://gitlab.com/gitlab-org/gitlab-ce/commit/12784559316fc2c3860641bee9c6f8e2bc12c4ba
Edited by Moritz OnkenMentioned in issue #22629 (closed)
-
@cluxter Thanks for giving everyone a summary of what worked for you. I'm not aware of anything that changed from 8.11 to 8.12 that would suddenly cause this problem (the current routing to gitlab-workhorse has been there for some time), but perhaps something else triggered it.
@filipa No, this does not look like a frontend-specific issue; it is more likely a configuration problem. I've removed the label.
I have just done a fresh install of gitlab on a new CentOS 7 machine, and have the same issue. I can see that the css file is present at /opt/gitlab/embedded/service/gitlab-rails/public/assets/application-1840f43fd20c30180b0599c214ddae7e6939d50933870904ebd0e81704658333.css however.
I'm just using the defaults and browsing through http://127.0.0.1:8080/ - which I presume is the nginx server but I really don't know as I'm that new to this! I downloaded from https://about.gitlab.com/downloads/#centos7 and followed the instructions, running the first install command given: curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash sudo yum install gitlab-ce
Does anyone have a fix for this yet, or is there maybe an older version that works so I can at least get up and running?
Thanks
-
@Matt-P Does http://127.0.0.1:8181 work?
@brianodonnell : can you link your config?
-
Apache 2.4
<VirtualHost *:443> ServerName gitlab.xxx.xxx ServerSignature Off ProxyPreserveHost On AllowEncodedSlashes NoDecode <Location /> Require all granted ProxyPassReverse http://127.0.0.1:8181 ProxyPassReverse https://gitlab.xxx.xxx/ </Location> RewriteEngine on RewriteCond %{REQUEST_URI} ^/api/v3/.* RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE] RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR] RewriteCond %{REQUEST_URI} ^/uploads/.* RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA] DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public RequestHeader set X_FORWARDED_PROTO 'https' RequestHeader set X-Forwarded-Ssl on ErrorDocument 404 /404.html ErrorDocument 422 /422.html ErrorDocument 500 /500.html ErrorDocument 503 /deploy.html SSLEngine On SSLCertificateFile /etc/pki/tls/certs/gitlab_xxx_xxx_cert.cer SSLCertificateKeyFile /etc/pki/tls/private/gitlab_xxx_xxx.key SSLCertificateChainFile /etc/pki/tls/certs/gitlab_xxx_xxx_interm.cer </VirtualHost> @brianodonnell : thanks
I can also confirm - it is resolved with the config above
Mentioned in issue #22698 (closed)
same here, works like a charm with @brianodonnell configs
Mentioned in issue #21929 (closed)
Could you point out what changes you made to your config to solve this issue? For example when I add the rewriting stuff I get an 503 instead.
Thanks!
Mine looks like this without any changes. I wonder what I need to add to solve this:
<VirtualHost *:443> SSLEngine on SSLCertificateFile /etc/apache2/sslcerts/gitlab.whatever.org.crt SSLCertificateKeyFile /etc/apache2/sslcerts/gitlab.whatever.org.private.key SSLCertificateChainFile /etc/apache2/sslcerts/gitlab.whatever.org.intermediate.crt ServerName gitlab.whatever.org ServerAdmin gitlab@whatever.org ErrorLog /var/log/apache2/gitlab_error.log CustomLog /var/log/apache2/gitlab_access.log combined ProxyRequests Off ProxyPreserveHost On ProxyPass / http://127.0.0.1:8080/ RequestHeader set X_FORWARDED_PROTO 'https' <Location /> ProxyPassReverse / Order deny,allow Allow from all </Location> </VirtualHost>-
@RalfEggert On line ProxyPass change port from 8080 to 8181 and it should work.
-
@RalfEggert What's the result of:
$ systemctl status gitlab-workhorse I don't know if this helps, but I did a
gitlab-ctl tailand reloaded the site. The css file starting withapplication-is processed by aProjectsControllerwhich results in the 401 error. The other asset files are now if I understand it correctly. Does this help to find a solution?==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/users/sign_in" for aaa.bbb.ccc.ddd at 2016-10-01 07:16:14 +0200 Processing by SessionsController#new as HTML Completed 200 OK in 27ms (Views: 7.7ms | ActiveRecord: 2.0ms) Started GET "/assets/application-481a0db50fbd5baaf710363e35d7ac792af08f710a842cf7e084cfd1cbe786d3.css" for aaa.bbb.ccc.ddd at 2016-10-01 07:16:14 +0200 Processing by ProjectsController#show as HTML Parameters: {"namespace_id"=>"assets", "id"=>"application-481a0db50fbd5baaf710363e35d7ac792af08f710a842cf7e084cfd1cbe786d3.css"} Completed 401 Unauthorized in 21ms (ActiveRecord: 2.1ms) Started GET "/assets/application-5fa120300b338070e8b574cfe457926b9cea7df9de1053d1b17f3439fb8b379f.js" for aaa.bbb.ccc.ddd at 2016-10-01 07:16:14 +0200 Started GET "/assets/print-68eed6d8135d858318821e790e25da27b2b4b9b8dbb1993fa6765d8e2e3e16ee.css" for aaa.bbb.ccc.ddd at 2016-10-01 07:16:14 +0200 Started GET "/users/sign_in" for aaa.bbb.ccc.ddd at 2016-10-01 07:16:15 +0200 Processing by SessionsController#new as HTML Completed 200 OK in 27ms (Views: 8.0ms | ActiveRecord: 2.0ms)-
@RalfEggert What is the result of:
# ps -ef | grep gitlab Here is the result:
root 1217 1 0 Jul25 ? 00:00:42 runsvdir -P /opt/gitlab/service log: ........................................................................................................................................................................................................................................................................................................................................................................................................... root 1227 1217 0 Jul25 ? 00:00:00 runsv gitlab-workhorse root 1234 1231 0 Jul25 ? 00:00:00 svlogd -tt /var/log/gitlab/logrotate root 1236 1229 0 Jul25 ? 00:00:03 svlogd -tt /var/log/gitlab/redis root 1237 1227 0 Jul25 ? 00:00:00 svlogd -tt /var/log/gitlab/gitlab-workhorse root 1238 1228 0 Jul25 ? 00:00:01 svlogd -tt /var/log/gitlab/sidekiq root 1239 1226 0 Jul25 ? 00:00:00 svlogd -tt /var/log/gitlab/unicorn root 1241 1230 0 Jul25 ? 00:00:00 svlogd -tt /var/log/gitlab/postgresql git 34366 1227 0 Okt01 ? 00:00:00 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/socket -authBackend http://localhost:8080 -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr -secretPath /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret gitlab-+ 34383 1230 0 Okt01 ? 00:00:01 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data gitlab-+ 34385 34383 0 Okt01 ? 00:00:00 postgres: checkpointer process gitlab-+ 34386 34383 0 Okt01 ? 00:00:01 postgres: writer process gitlab-+ 34387 34383 0 Okt01 ? 00:00:00 postgres: wal writer process gitlab-+ 34388 34383 0 Okt01 ? 00:00:02 postgres: autovacuum launcher process gitlab-+ 34389 34383 0 Okt01 ? 00:00:04 postgres: stats collector process gitlab-+ 34391 1229 0 Okt01 ? 00:04:47 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0 git 34397 1228 0 Okt01 ? 00:13:53 sidekiq 4.1.4 gitlab-rails [0 of 25 busy] git 34413 1226 0 Okt01 ? 00:00:55 /bin/bash /opt/gitlab/embedded/bin/gitlab-unicorn-wrapper git 34428 1 0 Okt01 ? 00:00:13 unicorn master -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru gitlab-+ 34440 34383 0 Okt01 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle gitlab-+ 34441 34383 0 Okt01 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle git 34442 34428 0 Okt01 ? 00:00:03 unicorn worker[0] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru git 34444 34428 0 Okt01 ? 00:00:03 unicorn worker[1] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru git 34447 34428 0 Okt01 ? 00:00:03 unicorn worker[2] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru gitlab-+ 34498 34383 0 Okt01 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle gitlab-+ 34500 34383 0 Okt01 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle gitlab-+ 34501 34383 0 Okt01 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle gitlab-+ 35631 34383 0 Okt01 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle root 52542 1231 0 10:32 ? 00:00:00 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper gitlab-+ 53624 34383 0 Okt01 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle root 55499 55428 0 11:14 pts/0 00:00:00 grep gitlabDoes this help to identify the issue?
-
@RalfEggert It totally does. Following what have been discussed in this thread, you need to use (and hence connect to)
gitlab-workhorseinstead ofgitlab-unicorn. The line with process34366shows thatgitlab-workhorsehas been launched with these parameters (among others):-listenNetwork unix -listenAddr /var/opt/gitlab/gitlab-workhorse/socketAccording to the
gitlab-workhorsemanual, this means that yourgitlab-workhorseis listening on a UNIX socket instead of a TCP/IP address. So find the Ubuntu init script that launches thegitlab-workhorseservice (I don't know anything about the Ubuntu init system), and use these parameters instead:-listenNetwork tcp -listenAddr 127.0.0.1:8181BTW these are the default settings, it should work the same if you just delete these 2 parameters. However this means that you'll have to configure a reverse proxy like Apache or nginx on the same machine to point to 127.0.0.1:8181, because 127.0.0.1 is only accessible from the local machine. If you want to make
gitlab-workhorsedirectly accessible from you LAN and/or NAT, use something like this (assuming your local IP address is 192.168.0.1):-listenNetwork tcp -listenAddr 192.168.0.1:8181Then you will be able to connect to
gitlab-workhorseusing http://192.168.0.1:8181 from your LAN and redirect ports on your NAT to this IP:port pair.Edited by Baptiste Rebillard @cluxter
Thanks for your kind help. I do feel still lost though because I have no idea where and how to change the configuration.
I did found the
/opt/gitlab/service/gitlab-workhorse/runfile which configureslistenNetworktounixandlistenAddrto/var/opt/gitlab/gitlab-workhorse/socket. I could change the values, but this file is overwritten by everygitlab-ctl reconfigurerun.In the
/etc/gitlab/gitlab.rbconfiguration file there is no mentioning ofgitlab-workhorseat all. All lines with the configuration for the mentionedunicornare commented out completely.So, I have no idea how to change it. Any further help is really appreciated.
-
If there is no mentioning of
gitlab-workhorsein/etc/gitlab/gitlab.rbthen we need to add something ;-)Paragraph 4 from https://docs.gitlab.com/omnibus/settings/nginx.html#using-a-non-bundled-web-server :
Add these lines to your
/etc/gitlab/gitlab.rbconfig file:gitlab_workhorse['listen_network'] = "tcp" gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"Then run:
sudo gitlab-ctl reconfigureYou should now be able to access
gitlab-workhorsethrough http://127.0.0.1:8181 . Funny enough I just stumbled about the same lines and wanted to add them just before you replied. After reconfiguration the settings in
/opt/gitlab/service/gitlab-workhorse/runhave been overwritten as expected. But still it did not help yet.I guess I need to do the reverse proxy like Apache you mentioned and again I have no idea how to set that up. Need to dig in again. Found this but not sure if that is the correct one:
Mentioned in issue #3225 (closed)
We use Apache as proxy and proxied to rails instead of workhorse. proxying to workhorse and starting workhorse solved this problem. It also solved a similar issue about not being able to edit .MD files
- See also gitlab-recipes#33 (closed) for configuring workhorse
- See https://gitlab.com/gitlab-org/gitlab-ce/issues/3225#note_15863024 for a similar issue with the ACE editor.
Same issue here, while upgrading from 8.10 to 8.12.4. We use apache proxying to rails. Fixed with what @cluxter is mentioning above (configure
gitlab-workhorseto listen to a port and redirect apache to that port instead of rails). Thank you!-)Going forward I think any new installation should create an (updated)
gitlab.rb.new, so that we have the always updated config options.Edited by Ioannis DrositisI can't believe i wasted so much time on this. I installed Gitlab for the first time (on fresh Ubuntu) and I saw new process listening on 8080. I thought "OK, that must be Gitlab" and it is Gitlab, but on 8080 you get this weird problems with CSS. The actual Gitlab with proper working CSS is on nginx, port 80 :/
-
@itsJoKr The actual GitLab is really on
gitlab-workhorseon port8181. Milestone changed to %8.13
Hoping that this will help someone. based on CentOS 7 Had a similar issue with the css on the sign in page giving a 404 error.
I have run this prior to an upgrade and the upgrade is then successful. I have then upgraded to another version again to ensure that is is carried through.
Note: this may not be considered good practice to rebuild the assets with an omnibus install.
Using feedback on several bugs logged with similar errors, I managed to fix it using the following method.
This has also been run after the upgrade which may also need a restart once carried out.
This is based on an omnibus install
#### Clear all assets known currently # gitlab-rake assets:clean ##### Ensure git has the access required to the directory # setfacl -R -m u:git:rwX /opt/gitlab/embedded/service/gitlab-rails/public/assets/ ##### Rebuild all the assets - change the ENV if yours does not match # gitlab-rake assets:precompile RAILS_ENV=production ##### give all users read and write to that folder # chmod -R a+rX /opt/gitlab/embedded/service/gitlab-rails/public/assets/ #### Remove git's write access # setfacl -R -x u:git /opt/gitlab/embedded/service/gitlab-rails/public/assets/ If a restart is required ## restart services using gitlab control # gitlab-ctl restart # restart the whole gitlab service tree at the OS level (Centos) # systemctl restart gitlab-runsvdir
Edited by carpnutter@carpnutter That fixed it for our machines! We've been having this issue on seemingly random machines and haven't been able to upgrade past 8.15.2 because of it.
Gitlab CE Omnibus 8.16.1 on Ubuntu 16.04 LTS
I'm going crazy! Nothing works for me... :( I've read so many threads in so many forums ...!
I have a nginx server which manages websites via subdirectories ( location /foo {...} and location /bar {...}). The server is behind a dns at home and all sites are working with ssl perfectly.
my gitlab-server (example ip: 192.168.1.50) is next to the webserver in the same network. The webserver should forward to the gitlab-server. Forwarding works. Only the assets files do not work on
https://www.example.com/git. I get the said 404 errors.https://www.example.com/git/assets/application-ae3d2248147aac8791dc1f09101eb43cdfa299927e48966da8647d6730aec123.csshttps://www.example.com/git/assets/application-92f2367dac38e643e4b853c819839b74b65e80c93de4b5b9aa3066ef8898abe7.jshttps://www.example.com/git/assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.cssI have done the following, but without success:
https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab https://docs.gitlab.com/omnibus/settings/configuration.html#relative-url-troubleshooting
nginx conf of my webserver:
server { listen 80 default_server; listen [::]:80 default_server; server_name example.com www.example.com; return 301 https://$server_name$request_uri; } server { listen 443 ssl default_server; listen [::]:443 ssl default_server; include snippets/ssl-example.com.conf; include snippets/ssl-params.conf; server_name example.com www.example.com; ssl on; ssl_protocols SSLv3; location /foo { ... } location /bar { ... } location /git { proxy_pass http://192.168.1.50; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Ssl on; proxy_set_header X-Forwarded-Port 443; } ... }My gitlab.rb:
external_url 'https://www.example.com/git' nginx['listen_port'] = 80 nginx['listen_https'] = false nginx['proxy_set_headers'] = { "X-Forwarded-Proto" => "https", "CUSTOM_HEADER" => "VALUE" }After setting the nginx webserver i restarted it with
systemctl reload nginx. after reconfiguring gitlab i reconfigured it and restarted it withgitlab-ctl reconfigureandgitlab-ctl restart.@eXorus Can you explain in detail how you got it?
Edited by bastianr-
Ok ... got it!
My baseline of the webserver is this tutorial:
https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-lemp-on-ubuntu-16-04
It´s really nice and easy. But you have to delete some lines in the explained nginx configuration!
... location = /favicon.ico { log_not_found off; access_log off; } location = /robots.txt { log_not_found off; access_log off; allow all; } location ~* \.(css|gif|ico|jpeg|jpg|js|png)$ { expires max; log_not_found off; } ...This helps me and resolved the issue. The other nginx and gitlab settings above are correct and currently in use.
After that I followed this tutorial to install gitlab non bundled.
So on, have fun! :)
Edited by bastianr I had the same problem. I tried @brianodonnell config and it works fine !!!
<VirtualHost *:80> ServerName git.example.com DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public ProxyPreserveHost On AllowEncodedSlashes NoDecode <Location /> Order deny,allow Allow from all Require all granted ProxyPassReverse http://127.0.0.1:8181 ProxyPassReverse http://git.example.com/ </Location> RewriteEngine on RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA] </VirtualHost>then
gitlab_workhorse['listen_network'] = "tcp" gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"Thanks a lot.
Whenever somebody is using the advanced config file listed here: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#vhost-server-block
Be-aware of the warning at the bottom (about "If you are using relative urls remove the block below"), this was working the in past. But broke during a migration from 8.16.2-ce to 8.16.6-c on Debian (omnibus installation, via separate Nginx server via unix socket).
So it should become:
#location ~ ^/(assets)/ { # root /opt/gitlab/embedded/service/gitlab-rails/public; # gzip_static on; # to serve pre-gzipped version # expires max; # add_header Cache-Control public; #}Restart Nginx server (sudo service nginx restart). I tried to set the assets directory permissions, rebuilding the assets, reconfigure, restarting.. well everything I tried. But commenting-out this block did work in my case.
Well I didn't know I used relative paths, but yea I did.. No idea how to make the paths absolute btw.
Edited by Melroy van den BergRecompiling assets appears to have been disabled in the omnibus version somewhere between 8.16.5 and 8.17.3 and now you'll get this error message when trying
rake aborted! ExecJS::Error: ExecJS disabled /opt/gitlab/embedded/bin/bundle:22:in `load' /opt/gitlab/embedded/bin/bundle:22:in `<main>' Tasks: TOP => assets:precompileI didn't setup our nginx configuration, but it certainly doesn't look like the advanced config. I may need to have a talk with the person who did. Looks like we're going to be on 8.16.5 for awhile.
@vilhelmen why are you trying to recompile assets? You shouldn't need to any more: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/configuration.md#relative-url-requirements
@smcgivern Whenever I upgrade some of our EE machines, the CSS stops working. I know the docs have said there should be no need to recompile on omnibus, but it fixes it, so I did it anyway.
We aren't using a relative url, but we (unfortunately) have multiple services on the same machine, so we use an external Nginx install to handle the subdomains. I didn't do the configuration for Nginx, but it appears to only have a / block that forwards everything to the workhorse socket, which seems roughly correct.
I could have sworn it happens on one machine with external nginx and one that is pure omnibus, and another that is pure omnibus is wholly unaffected. I haven't been keeping things as up to date as I should lately so I may be misremembering slightly.
Once classes are over for the day I can break a machine to scrape some logs if that will help, though I believe it looked a lot like those in the original post.
-
@vilhelmen thanks, that would be useful.
@mikegreiling should this task even be available in Omnibus installs?
@smcgivern Here's the tail from loading the sign in page
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/users/sign_in" for 128.206.116.250 at 2017-03-16 12:41:45 -0500 Processing by SessionsController#new as HTML Completed 200 OK in 140ms (Views: 70.6ms | ActiveRecord: 14.8ms) ==> /var/log/gitlab/gitlab-workhorse/current <== 2017-03-16_17:41:46.18454 git.dsa.missouri.edu @ - - [2017-03-16 12:41:45.841156552 -0500 CDT] "GET /users/sign_in HTTP/1.1" 200 11736 "https://git.dsa.missouri.edu/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" 0.343307 2017-03-16_17:41:46.23421 git.dsa.missouri.edu @ - - [2017-03-16 12:41:46.233959361 -0500 CDT] "GET /assets/application-a6dd150d84720bf9a3c0d83ce742846db842b2f38248e1dd91159801d5aa5f41.css HTTP/1.1" 404 19 "https://git.dsa.missouri.edu/users/sign_in" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" 0.000153 2017-03-16_17:41:46.23507 2017/03/16 12:41:46 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.css" ("gzip") for GET "/assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.css" 2017-03-16_17:41:46.24735 git.dsa.missouri.edu @ - - [2017-03-16 12:41:46.234860011 -0500 CDT] "GET /assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.css HTTP/1.1" 200 350 "https://git.dsa.missouri.edu/users/sign_in" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" 0.012422 2017-03-16_17:41:46.28646 2017/03/16 12:41:46 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/application-70c67ed9d057a03a61dd-v2.js" ("gzip") for GET "/assets/webpack/application-70c67ed9d057a03a61dd-v2.js" 2017-03-16_17:41:46.28757 git.dsa.missouri.edu @ - - [2017-03-16 12:41:46.2862999 -0500 CDT] "GET /assets/webpack/application-70c67ed9d057a03a61dd-v2.js HTTP/1.1" 200 269494 "https://git.dsa.missouri.edu/users/sign_in" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" 0.001194 ==> /var/log/gitlab/redis/current <== 2017-03-16_17:41:47.08736 74734:M 16 Mar 12:41:47.087 * 10000 changes in 60 seconds. Saving... 2017-03-16_17:41:47.08812 74734:M 16 Mar 12:41:47.088 * Background saving started by pid 75560 2017-03-16_17:41:47.09201 75560:C 16 Mar 12:41:47.091 * DB saved on disk 2017-03-16_17:41:47.09245 75560:C 16 Mar 12:41:47.092 * RDB: 6 MB of memory used by copy-on-write ==> /var/log/gitlab/gitlab-workhorse/current <== 2017-03-16_17:41:47.17153 2017/03/16 12:41:47 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico" ("gzip") for GET "/assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico" 2017-03-16_17:41:47.17162 git.dsa.missouri.edu @ - - [2017-03-16 12:41:47.171329345 -0500 CDT] "GET /assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico HTTP/1.1" 200 1384 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" 0.000232 ==> /var/log/gitlab/redis/current <== 2017-03-16_17:41:47.18890 74734:M 16 Mar 12:41:47.188 * Background saving terminated with successThe console in Firefox always reports a MIME issue when this happens, but I'm not sure it's really valid. How can a type mismatch if you never received the file? Maybe it's trying to interpret the 404 page as the CSS.
GET https://git.dsa.missouri.edu/assets/application-a6dd150d84720bf9a3c0d83ce742846db842b2f38248e1dd91159801d5aa5f41.css [HTTP/1.1 404 Not Found 1ms] The resource from “https://git.dsa.missouri.edu/assets/application-a6dd150d84720bf9a3c0d83ce742846db842b2f38248e1dd91159801d5aa5f41.css” was blocked due to MIME type mismatch (X-Content-Type-Options: nosniff).Rake check doesn't reveal anything, but I'll attach it anyway. rake_check.txt
-
@vilhelmen as far as I know the asset exists under that filename in the package itself. @mikegreiling is that right?
@smcgivern Cloned a different machine with the same issue so we can pick it apart if need be, here's the log tail as well as the contents of /opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/" for 172.16.20.50 at 2017-03-17 12:40:25 -0500 Processing by RootController#index as HTML Redirected to https://172.16.11.168/users/sign_in Filter chain halted as :redirect_unlogged_user rendered or redirected Completed 302 Found in 20ms (ActiveRecord: 1.6ms) ==> /var/log/gitlab/gitlab-workhorse/current <== 2017-03-17_17:40:25.34483 172.16.11.168 @ - - [2017-03-17 12:40:25.313959779 -0500 CDT] "GET / HTTP/1.1" 302 101 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" 0.030805 ==> /var/log/gitlab/nginx/gitlab_access.log <== 172.16.20.50 - - [17/Mar/2017:12:40:25 -0500] "GET / HTTP/2.0" 302 388 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" ==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/users/sign_in" for 172.16.20.50 at 2017-03-17 12:40:25 -0500 Processing by SessionsController#new as HTML Completed 200 OK in 43ms (Views: 17.7ms | ActiveRecord: 3.8ms) ==> /var/log/gitlab/gitlab-workhorse/current <== 2017-03-17_17:40:25.39413 172.16.11.168 @ - - [2017-03-17 12:40:25.34765579 -0500 CDT] "GET /users/sign_in HTTP/1.1" 200 8154 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" 0.046407 ==> /var/log/gitlab/nginx/gitlab_access.log <== 172.16.20.50 - - [17/Mar/2017:12:40:25 -0500] "GET /users/sign_in HTTP/2.0" 200 8458 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" ==> /var/log/gitlab/gitlab-workhorse/current <== 2017-03-17_17:40:25.39920 172.16.11.168 @ - - [2017-03-17 12:40:25.399042957 -0500 CDT] "GET /assets/application-a6dd150d84720bf9a3c0d83ce742846db842b2f38248e1dd91159801d5aa5f41.css HTTP/1.1" 404 19 "https://172.16.11.168/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" 0.000095 2017-03-17_17:40:25.39926 2017/03/17 12:40:25 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/application-70c67ed9d057a03a61dd-v2.js" ("gzip") for GET "/assets/webpack/application-70c67ed9d057a03a61dd-v2.js" ==> /var/log/gitlab/nginx/gitlab_access.log <== 172.16.20.50 - - [17/Mar/2017:12:40:25 -0500] "GET /assets/application-a6dd150d84720bf9a3c0d83ce742846db842b2f38248e1dd91159801d5aa5f41.css HTTP/2.0" 404 107 "https://172.16.11.168/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" ==> /var/log/gitlab/gitlab-workhorse/current <== 2017-03-17_17:40:25.40109 172.16.11.168 @ - - [2017-03-17 12:40:25.399137624 -0500 CDT] "GET /assets/webpack/application-70c67ed9d057a03a61dd-v2.js HTTP/1.1" 200 269494 "https://172.16.11.168/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" 0.001857 ==> /var/log/gitlab/nginx/gitlab_access.log <== 172.16.20.50 - - [17/Mar/2017:12:40:25 -0500] "GET /assets/webpack/application-70c67ed9d057a03a61dd-v2.js HTTP/2.0" 200 269650 "https://172.16.11.168/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" ==> /var/log/gitlab/gitlab-workhorse/current <== 2017-03-17_17:40:25.43261 2017/03/17 12:40:25 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.css" ("gzip") for GET "/assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.css" 2017-03-17_17:40:25.43270 172.16.11.168 @ - - [2017-03-17 12:40:25.432499501 -0500 CDT] "GET /assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.css HTTP/1.1" 200 350 "https://172.16.11.168/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" 0.000138 ==> /var/log/gitlab/nginx/gitlab_access.log <== 172.16.20.50 - - [17/Mar/2017:12:40:25 -0500] "GET /assets/print-9c3a1eb4a2f45c9f3d7dd4de03f14c2e6b921e757168b595d7f161bbc320fc05.css HTTP/2.0" 200 528 "https://172.16.11.168/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36"[root@corona-dbg webpack]# ll total 21016 -rw-r--r--. 1 root root 945194 Mar 7 16:09 application-70c67ed9d057a03a61dd-v2.js -rw-r--r--. 1 root root 269494 Mar 7 16:09 application-70c67ed9d057a03a61dd-v2.js.gz -rw-r--r--. 1 root root 5512928 Mar 7 16:09 application-70c67ed9d057a03a61dd-v2.js.map -rw-r--r--. 1 root root 1372152 Mar 7 16:09 application-70c67ed9d057a03a61dd-v2.js.map.gz -rw-r--r--. 1 root root 2753 Mar 7 16:09 blob_edit-015d33cb3fb9d27ec767-v2.js -rw-r--r--. 1 root root 1104 Mar 7 16:09 blob_edit-015d33cb3fb9d27ec767-v2.js.gz -rw-r--r--. 1 root root 17219 Mar 7 16:09 blob_edit-015d33cb3fb9d27ec767-v2.js.map -rw-r--r--. 1 root root 4143 Mar 7 16:09 blob_edit-015d33cb3fb9d27ec767-v2.js.map.gz -rw-r--r--. 1 root root 146615 Mar 7 16:09 boards-662e551ffc94960e61fe-v2.js -rw-r--r--. 1 root root 45778 Mar 7 16:09 boards-662e551ffc94960e61fe-v2.js.gz -rw-r--r--. 1 root root 639844 Mar 7 16:09 boards-662e551ffc94960e61fe-v2.js.map -rw-r--r--. 1 root root 177064 Mar 7 16:09 boards-662e551ffc94960e61fe-v2.js.map.gz -rw-r--r--. 1 root root 2225 Mar 7 16:09 boards_test-990daf654abf8eb792c5-v2.js -rw-r--r--. 1 root root 1098 Mar 7 16:09 boards_test-990daf654abf8eb792c5-v2.js.gz -rw-r--r--. 1 root root 17002 Mar 7 16:09 boards_test-990daf654abf8eb792c5-v2.js.map -rw-r--r--. 1 root root 4312 Mar 7 16:09 boards_test-990daf654abf8eb792c5-v2.js.map.gz -rw-r--r--. 1 root root 108347 Mar 7 16:09 commit_pipelines-bc2b628333c1173b4ed6-v2.js -rw-r--r--. 1 root root 36241 Mar 7 16:09 commit_pipelines-bc2b628333c1173b4ed6-v2.js.gz -rw-r--r--. 1 root root 470370 Mar 7 16:09 commit_pipelines-bc2b628333c1173b4ed6-v2.js.map -rw-r--r--. 1 root root 142111 Mar 7 16:09 commit_pipelines-bc2b628333c1173b4ed6-v2.js.map.gz -rw-r--r--. 1 root root 99574 Mar 7 16:09 cycle_analytics-5feed6bcfb6646b7e987-v2.js -rw-r--r--. 1 root root 32505 Mar 7 16:09 cycle_analytics-5feed6bcfb6646b7e987-v2.js.gz -rw-r--r--. 1 root root 374908 Mar 7 16:09 cycle_analytics-5feed6bcfb6646b7e987-v2.js.map -rw-r--r--. 1 root root 112922 Mar 7 16:09 cycle_analytics-5feed6bcfb6646b7e987-v2.js.map.gz -rw-r--r--. 1 root root 90615 Mar 7 16:09 diff_notes-6ed778a5a0d5e30b8acb-v2.js -rw-r--r--. 1 root root 31057 Mar 7 16:09 diff_notes-6ed778a5a0d5e30b8acb-v2.js.gz -rw-r--r--. 1 root root 392193 Mar 7 16:09 diff_notes-6ed778a5a0d5e30b8acb-v2.js.map -rw-r--r--. 1 root root 116991 Mar 7 16:09 diff_notes-6ed778a5a0d5e30b8acb-v2.js.map.gz -rw-r--r--. 1 root root 113364 Mar 7 16:09 environments-17f6d709b1764175f9d5-v2.js -rw-r--r--. 1 root root 37560 Mar 7 16:09 environments-17f6d709b1764175f9d5-v2.js.gz -rw-r--r--. 1 root root 491647 Mar 7 16:09 environments-17f6d709b1764175f9d5-v2.js.map -rw-r--r--. 1 root root 143609 Mar 7 16:09 environments-17f6d709b1764175f9d5-v2.js.map.gz -rw-r--r--. 1 root root 26436 Mar 7 16:09 filtered_search-f002d80f212885c0b347-v2.js -rw-r--r--. 1 root root 5945 Mar 7 16:09 filtered_search-f002d80f212885c0b347-v2.js.gz -rw-r--r--. 1 root root 128471 Mar 7 16:09 filtered_search-f002d80f212885c0b347-v2.js.map -rw-r--r--. 1 root root 27947 Mar 7 16:09 filtered_search-f002d80f212885c0b347-v2.js.map.gz -rw-r--r--. 1 root root 163572 Mar 7 16:09 graphs-50167501d51321b9add8-v2.js -rw-r--r--. 1 root root 56114 Mar 7 16:09 graphs-50167501d51321b9add8-v2.js.gz -rw-r--r--. 1 root root 1035693 Mar 7 16:09 graphs-50167501d51321b9add8-v2.js.map -rw-r--r--. 1 root root 242483 Mar 7 16:09 graphs-50167501d51321b9add8-v2.js.map.gz -rw-r--r--. 1 root root 15121 Mar 7 16:09 issuable-4395f065dec7b1377787-v2.js -rw-r--r--. 1 root root 3610 Mar 7 16:09 issuable-4395f065dec7b1377787-v2.js.gz -rw-r--r--. 1 root root 67230 Mar 7 16:09 issuable-4395f065dec7b1377787-v2.js.map -rw-r--r--. 1 root root 13121 Mar 7 16:09 issuable-4395f065dec7b1377787-v2.js.map.gz -rw-r--r--. 1 root root 52930 Mar 7 16:09 lib_chart-3b350422be827e3c8b71-v2.js -rw-r--r--. 1 root root 11970 Mar 7 16:09 lib_chart-3b350422be827e3c8b71-v2.js.gz -rw-r--r--. 1 root root 320201 Mar 7 16:09 lib_chart-3b350422be827e3c8b71-v2.js.map -rw-r--r--. 1 root root 68268 Mar 7 16:09 lib_chart-3b350422be827e3c8b71-v2.js.map.gz -rw-r--r--. 1 root root 152231 Mar 7 16:09 lib_d3-5a909fc0117033fbe705-v2.js -rw-r--r--. 1 root root 53492 Mar 7 16:09 lib_d3-5a909fc0117033fbe705-v2.js.gz -rw-r--r--. 1 root root 975144 Mar 7 16:09 lib_d3-5a909fc0117033fbe705-v2.js.map -rw-r--r--. 1 root root 229011 Mar 7 16:09 lib_d3-5a909fc0117033fbe705-v2.js.map.gz -rw-r--r--. 1 root root 77389 Mar 7 16:09 lib_vue-39b983fba26af5199758-v2.js -rw-r--r--. 1 root root 27860 Mar 7 16:09 lib_vue-39b983fba26af5199758-v2.js.gz -rw-r--r--. 1 root root 311898 Mar 7 16:09 lib_vue-39b983fba26af5199758-v2.js.map -rw-r--r--. 1 root root 99759 Mar 7 16:09 lib_vue-39b983fba26af5199758-v2.js.map.gz -rw-r--r--. 1 root root 20271 Mar 7 16:09 manifest.json -rw-r--r--. 1 root root 76936 Mar 7 16:09 merge_conflicts-5aca6c48c77ac46f4115-v2.js -rw-r--r--. 1 root root 27061 Mar 7 16:09 merge_conflicts-5aca6c48c77ac46f4115-v2.js.gz -rw-r--r--. 1 root root 295167 Mar 7 16:09 merge_conflicts-5aca6c48c77ac46f4115-v2.js.map -rw-r--r--. 1 root root 95358 Mar 7 16:09 merge_conflicts-5aca6c48c77ac46f4115-v2.js.map.gz -rw-r--r--. 1 root root 1760 Mar 7 16:09 merge_request_widget-8f66cd055bd5899a1cfa-v2.js -rw-r--r--. 1 root root 718 Mar 7 16:09 merge_request_widget-8f66cd055bd5899a1cfa-v2.js.gz -rw-r--r--. 1 root root 11461 Mar 7 16:09 merge_request_widget-8f66cd055bd5899a1cfa-v2.js.map -rw-r--r--. 1 root root 2798 Mar 7 16:09 merge_request_widget-8f66cd055bd5899a1cfa-v2.js.map.gz -rw-r--r--. 1 root root 87121 Mar 7 16:09 mr_widget_ee-092cf0edd691a2a06d75-v2.js -rw-r--r--. 1 root root 30039 Mar 7 16:09 mr_widget_ee-092cf0edd691a2a06d75-v2.js.gz -rw-r--r--. 1 root root 352270 Mar 7 16:09 mr_widget_ee-092cf0edd691a2a06d75-v2.js.map -rw-r--r--. 1 root root 109111 Mar 7 16:09 mr_widget_ee-092cf0edd691a2a06d75-v2.js.map.gz -rw-r--r--. 1 root root 9017 Mar 7 16:09 network-26aff28014e5c3396a85-v2.js -rw-r--r--. 1 root root 3135 Mar 7 16:09 network-26aff28014e5c3396a85-v2.js.gz -rw-r--r--. 1 root root 53423 Mar 7 16:09 network-26aff28014e5c3396a85-v2.js.map -rw-r--r--. 1 root root 11954 Mar 7 16:09 network-26aff28014e5c3396a85-v2.js.map.gz -rw-r--r--. 1 root root 7991 Mar 7 16:09 profile-1b5e086be9f380745af0-v2.js -rw-r--r--. 1 root root 2714 Mar 7 16:09 profile-1b5e086be9f380745af0-v2.js.gz -rw-r--r--. 1 root root 41113 Mar 7 16:09 profile-1b5e086be9f380745af0-v2.js.map -rw-r--r--. 1 root root 9680 Mar 7 16:09 profile-1b5e086be9f380745af0-v2.js.map.gz -rw-r--r--. 1 root root 15260 Mar 7 16:09 protected_branches-b0453100e933de3cbc85-v2.js -rw-r--r--. 1 root root 4183 Mar 7 16:09 protected_branches-b0453100e933de3cbc85-v2.js.gz -rw-r--r--. 1 root root 87158 Mar 7 16:09 protected_branches-b0453100e933de3cbc85-v2.js.map -rw-r--r--. 1 root root 19201 Mar 7 16:09 protected_branches-b0453100e933de3cbc85-v2.js.map.gz -rw-r--r--. 1 root root 974 Mar 7 16:09 snippet-11fe6c9225a081b78d02-v2.js -rw-r--r--. 1 root root 542 Mar 7 16:09 snippet-11fe6c9225a081b78d02-v2.js.gz -rw-r--r--. 1 root root 8973 Mar 7 16:09 snippet-11fe6c9225a081b78d02-v2.js.map -rw-r--r--. 1 root root 2339 Mar 7 16:09 snippet-11fe6c9225a081b78d02-v2.js.map.gz -rw-r--r--. 1 root root 604357 Mar 7 16:09 terminal-90c9d241fa9298c28b0e-v2.js -rw-r--r--. 1 root root 210611 Mar 7 16:09 terminal-90c9d241fa9298c28b0e-v2.js.gz -rw-r--r--. 1 root root 1652629 Mar 7 16:09 terminal-90c9d241fa9298c28b0e-v2.js.map -rw-r--r--. 1 root root 520588 Mar 7 16:09 terminal-90c9d241fa9298c28b0e-v2.js.map.gz -rw-r--r--. 1 root root 5687 Mar 7 16:09 users-e8a391b905ba4224d868-v2.js -rw-r--r--. 1 root root 2032 Mar 7 16:09 users-e8a391b905ba4224d868-v2.js.gz -rw-r--r--. 1 root root 33008 Mar 7 16:09 users-e8a391b905ba4224d868-v2.js.map -rw-r--r--. 1 root root 7499 Mar 7 16:09 users-e8a391b905ba4224d868-v2.js.map.gz -rw-r--r--. 1 root root 109912 Mar 7 16:09 vue_pipelines-d7db90bf895a28ab91d2-v2.js -rw-r--r--. 1 root root 36995 Mar 7 16:09 vue_pipelines-d7db90bf895a28ab91d2-v2.js.gz -rw-r--r--. 1 root root 476473 Mar 7 16:09 vue_pipelines-d7db90bf895a28ab91d2-v2.js.map -rw-r--r--. 1 root root 143446 Mar 7 16:09 vue_pipelines-d7db90bf895a28ab91d2-v2.js.map.gz-
@vilhelmen CSS is in the parent folder. What does that contain?
@smcgivern I was close! Listing attached. The a6 application file doesn't appear to be there. assets.txt
-
@vilhelmen which package is that from? That doesn't match the 8.17.3 CE or EE packages, AFAICT.
@smcgivern Well I suppose that explains the issue. It's 8.17.3 as far as yum is concerned. I just noticed that the timestamps don't match the install time (March 17th), is the 7th when 8.17.3 was released/built?
Installed Packages Name : gitlab-ee Arch : x86_64 Version : 8.17.3 Release : ee.0.el7 Size : 858 M Repo : installed From repo : gitlab_gitlab-ee Summary : GitLab Enterprise Edition and GitLab CI (including NGINX, : Postgres, Redis) URL : https://about.gitlab.com/ License : MIT Description : GitLab Enterprise Edition and GitLab CI (including NGINX, : Postgres, Redis)-
@vilhelmen OK, so that's this package: https://packages.gitlab.com/gitlab/gitlab-ee/packages/el/7/gitlab-ee-8.17.3-ee.0.el7.x86_64.rpm?
We did release it on 7 March, yes. And I'm an idiot - I was looking at the Debian package, but it does actually contain the same CSS file as your installation
-
@vilhelmen as you're using EE, would you mind emailing support for this? I tried - and failed - to help here, but they are much better at this than I am
@smcgivern Thanks for the help anyway, I really appreciate it! Time to finally figure out where the EE creds are.
@smcgivern Had a nice debugging session with support today. Short version: No one knows what the issue is.
We were, however, able to fix it by purging the gitlab install and restoring from a backup, which I suppose was the nuclear option. So, if you're using omnibus and are not using a relative path and having the problem, a purge/restore may be easiest. It's actually pretty simple, at least with our basic setup.
Here's how I fixed our Cent7 machines:
- Run a backup
gitlab-rake gitlab:backup:create - Move the backup from
/var/opt/gitlab/backups - Copy the
/etc/gitlabdirectory - Run the gitlab uninstaller
gitlab-ctl uninstall(Make sure the etc dir and the backup have been moved) - Stop/disable/kill the systemd service and anything still running (this may be better before the uninstall)
systemctl stop gitlab-runsvdir.service - Uninstall gitlab
yum erase gitlab-ee(make sure you remember what version you are on) - Delete all remaining gitlab data (you probably don't have to wipe the log dir)
rm -rf /var/opt/gitlab /var/log/gitlab /etc/gitlab /opt/gitlab - Reinstall whatever version you were on
yum install gitlab-ee-VERSION - Recheck/start/enable the gitlab systemd service
systemctl daemon-reload,systemctl enable gitlab-runsvdir.service, andsystemctl start gitlab-runsvdir.service - Edit the
/etc/gitlab/gitlab.rband configure the external url. Reconfigure gitlabgitlab-ctl reconfigureand set the root password in the browser. - Wipe the
/etc/gitlabdirectory and replace it with the backup from earlier. Reconfigure gitlab again.gitlab-ctl reconfigure - Copy the backup back to
/var/opt/gitlab/backups/, chown it to git:git, and copy the timestamp string (looks likeEPOCH_YYYY_MM_DD). - Stop unicorn and sidekiq services.
gitlab-ctl stop unicorngitlab-ctl stop sidekiq - Restore backup with
gitlab-rake gitlab:backup:restore BACKUP=BACKUP_TIMESTAMP_STRING, answer yes to all three(?) prompts - Restart gitlab
gitlab-ctl restart
I was able to update with no issues after that. It's easier than it looks. Here's the backup documentation if you're not using the omnibus install https://docs.gitlab.com/ce/raketasks/backup_restore.html
- Run a backup
-
Awesome @vilhelmen, I'm glad it's fixed now!
I have the issue now, on version 11.11. I rare use gitlab (may be once per few month), but it auto update every day. Centos 7, omnibus.
I don't know exact time/version when issue start.
It was different names to application-.css in /opt/gitlab/embedded/service/gitlab-rails/public/assets/application-.css and to load from frontend.
The https://gitlab.com/gitlab-org/gitlab-ce/issues/22528#note_27223043 - help.
Before it I tried: gitlab-ctl reconfigure, gitlab-rake tmp:cache:clear, gitlab-rake cache:clear, gitlab-ctl reconfigure
it didn't help.
Edited by rekby
Thanks for the summary @vilhelmen I followed this on my Debian server (with some minor changes) and I was finally able to fully repair my GitLab instance!
EDIT: I wrote a blog post for the Debian version. I couldn't find a way to contact you directly to ask for your permission, if you've got any issue please let me know!
Edited by Outpoxmentioned in issue #28630 (closed)
-
Thanks @vilhelmen! See my comment in https://gitlab.com/gitlab-org/gitlab-ce/issues/28630#note_27837124.
mentioned in issue #31411 (closed)
@stanhu I moved the
application.scssfile to my home directory, then ran a reconfigure & restart, but I am still experiencing the 404 error.I also tried running a script which worked previously in 8.x:
setfacl -R -m u:git:rwX /opt/gitlab/embedded/service/gitlab-rails/public/assets/ gitlab-rake assets:clean rm /opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME gitlab-rake assets:precompile RAILS_ENV=production chmod -R a+rX /opt/gitlab/embedded/service/gitlab-rails/public/assets/ setfacl -R -x u:git /opt/gitlab/embedded/service/gitlab-rails/public/assets/ gitlab-ctl restart- Centos 7.0.1406
- gitlab-ee-9.1.0-ee.0.el7.x86_64 (omnibus)
This system was updated from prior release gitlab-ee-8.17.4-ee.0.el7.x86_64.
Edited by Dan LoweI ended up following @vilhelmen's procedure to fix my EE install on Centos 7. It worked well, and as a side effect, exposed the fact my backups had been broken due to low disk space, so I was able to fix that. Thanks!
