Unable to create a new project
Summary
I host ~40 projects on my own GitLab server for 2 months. Now I've figured out that I can not crate a new project any more. I've tried to solve the bug by upgrading Gitlab from 11.0.2 -> 11.3.5. It did't help. Below I'm describing the bug for the last version.
Steps to reproduce
- Click on 'New Project' on GitLab dashboard
- Fill 'Project name' field with 'NewProject'
- Click on 'Create project'
What is the current bug behavior?
The web site hangs for a while and then I get Error 502. The project is not created at all and I don't see it in the list of the project in Admin dashboard.
What is the expected correct behavior?
The project must have been created as all the former project I created before.
Relevant logs and/or screenshots
In the logs I see suspicions messages:
2018-10-17_04:59:40.92054 FATAL: terminating connection due to idle-in-transaction timeout
E, [2018-10-17T04:59:41.572845 #7252] ERROR -- : worker=3 PID:7360 timeout (61s > 60s), killing Output of checks
Results of GitLab environment info
Output of gitlab-rake gitlab:env:info:
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.4.4p296
Gem Version: 2.7.6
Bundler Version:1.16.2
Rake Version: 12.3.1
Redis Version: 3.2.11
Git Version: 2.18.1
Sidekiq Version:5.1.3
Go Version: unknown
GitLab information
Version: 11.3.5
Revision: c74e8d9
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://dras-rsu-ci:9000/gitlab
HTTP Clone URL: https://dras-rsu-ci:9000/gitlab/some-group/some-project.git
SSH Clone URL: git@dras-rsu-ci:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 8.3.3
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Output of gitlab-rake gitlab:check SANITIZE=true:
Checking GitLab Shell ...
GitLab Shell version >= 8.3.3 ? ... OK (8.3.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/1 ... ok
2/2 ... ok
2/3 ... ok
2/4 ... ok
6/5 ... ok
6/6 ... ok
6/7 ... ok
6/8 ... ok
6/9 ... ok
6/10 ... ok
6/11 ... ok
6/12 ... ok
7/13 ... ok
7/14 ... ok
7/15 ... ok
7/16 ... repository is empty
7/17 ... ok
7/18 ... ok
7/19 ... ok
8/20 ... ok
8/21 ... ok
8/22 ... ok
8/23 ... ok
18/26 ... ok
13/27 ... ok
13/28 ... ok
13/29 ... ok
13/30 ... ok
7/31 ... ok
13/32 ... ok
2/33 ... ok
8/34 ... ok
8/35 ... ok
15/36 ... ok
15/37 ... ok
7/38 ... ok
8/39 ... ok
18/40 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
2/3 ... yes
2/4 ... yes
6/5 ... yes
6/6 ... yes
6/7 ... yes
6/8 ... yes
6/9 ... yes
6/10 ... yes
6/11 ... yes
6/12 ... yes
7/13 ... yes
7/14 ... yes
7/15 ... yes
7/16 ... yes
7/17 ... yes
7/18 ... yes
7/19 ... yes
8/20 ... yes
8/21 ... yes
8/22 ... yes
8/23 ... yes
18/26 ... yes
13/27 ... yes
13/28 ... yes
13/29 ... yes
13/30 ... yes
7/31 ... yes
13/32 ... yes
2/33 ... yes
8/34 ... yes
8/35 ... yes
15/36 ... yes
15/37 ... yes
7/38 ... yes
8/39 ... yes
18/40 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.4)
Git version >= 2.9.5 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 13
Checking GitLab ... Finished
Possible fixes
I've tried to increase timeout settings in gitlab.rb:
gitlab_rails['gitlab_shell_git_timeout'] = 8000
unicorn['worker_timeout'] = 600It did not help me, but I got error 404 instead of 502
Activity
I seem to have the exact same problem. We have it for about 3 month now, i noticed it first early august, i know for sure it worked early june.
I get the same error message and i tried the same from your
Possible fixessection with the same result.I have my GitLab running in a docker container that I update every few weeks so I always thought a update broke it (so sometime between june and august), but as I understand it for you it worked at first with 11.0.2 and then it suddenly didnt?!
my container log from during the project create until the timeout:
web_1 | [195/1847] web_1 | ==> /var/log/gitlab/gitaly/current <== web_1 | 2018-10-17_09:24:08.28778 time="2018-10-17T09:24:08Z" level=info msg="finished unary call" grpc.code=OK grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-web grpc.method=NamespaceExists grpc.request.Name=test/blarg.git grpc.request.StorageName=default grpc. service=gitaly.NamespaceService grpc.time_ms=0 peer.address=@ span.kind=server system=grpc web_1 | 2018-10-17_09:24:08.28881 time="2018-10-17T09:24:08Z" level=info msg="finished unary call" grpc.code=OK grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-web grpc.method=AddNamespace grpc.request.Name=test grpc.request.StorageName=default grpc.service=gital y.NamespaceService grpc.time_ms=0 peer.address=@ span.kind=server system=grpc web_1 | 2018-10-17_09:24:08.30090 time="2018-10-17T09:24:08Z" level=info msg="finished unary call" grpc.code=OK grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-web grpc.method=AddNamespace grpc.request.Name=test grpc.request.StorageName=default grpc.service=gital y.NamespaceService grpc.time_ms=0 peer.address=@ span.kind=server system=grpc web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production.log <== web_1 | Started POST "/projects" for 192.168.10.128 at 2018-10-17 09:24:08 +0000 web_1 | Processing by ProjectsController#create as HTML web_1 | Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "project"=>{"ci_cd_only"=>"false", "name"=>"blarg", "namespace_id"=>"7", "path"=>"blarg", "description"=>""}} web_1 | web_1 | ==> /var/log/gitlab/gitlab-monitor/current <== web_1 | 2018-10-17_09:24:08.80096 ::1 - - [17/Oct/2018:09:24:08 UTC] "GET /process HTTP/1.1" 200 1501 web_1 | 2018-10-17_09:24:08.80097 - -> /process web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== web_1 | [2018-10-17 09:24:11] 127.0.0.1 - - [17/Oct/2018:09:24:11 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/1.8.2" web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production_json.log <== web_1 | {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":3.04,"view":0.35,"db":0.0,"time":"2018-10-17T09:24:15.826Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null} web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production.log <== web_1 | Started GET "/-/metrics" for 127.0.0.1 at 2018-10-17 09:24:15 +0000 web_1 | Processing by MetricsController#index as HTML web_1 | Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms) web_1 | web_1 | ==> /var/log/gitlab/gitlab-monitor/current <== web_1 | 2018-10-17_09:24:19.63082 ::1 - - [17/Oct/2018:09:24:19 UTC] "GET /sidekiq HTTP/1.1" 200 15458 web_1 | 2018-10-17_09:24:19.63084 - -> /sidekiq web_1 | 2018-10-17_09:24:21.53696 ::1 - - [17/Oct/2018:09:24:21 UTC] "GET /database HTTP/1.1" 200 66450 web_1 | 2018-10-17_09:24:21.53697 - -> /database web_1 | 2018-10-17_09:24:23.80034 ::1 - - [17/Oct/2018:09:24:23 UTC] "GET /process HTTP/1.1" 200 1514 web_1 | 2018-10-17_09:24:23.80035 - -> /process web_1 | web_1 | ==> /var/log/gitlab/alertmanager/current <== web_1 | 2018-10-17_09:24:24.74021 level=info ts=2018-10-17T09:24:24.740085451Z caller=cluster.go:579 component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=59.993876624s web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== web_1 | [2018-10-17 09:24:26] 127.0.0.1 - - [17/Oct/2018:09:24:26 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/1.8.2" web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production_json.log <== web_1 | {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.76,"view":0.31,"db":0.0,"time":"2018-10-17T09:24:30.825Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null} web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production.log <== web_1 | Started GET "/-/metrics" for 127.0.0.1 at 2018-10-17 09:24:30 +0000 web_1 | Processing by MetricsController#index as HTML web_1 | Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms) web_1 | web_1 | ==> /var/log/gitlab/gitlab-monitor/current <== web_1 | 2018-10-17_09:24:34.63000 ::1 - - [17/Oct/2018:09:24:34 UTC] "GET /sidekiq HTTP/1.1" 200 15458 web_1 | 2018-10-17_09:24:34.63004 - -> /sidekiq web_1 | 2018-10-17_09:24:36.53576 ::1 - - [17/Oct/2018:09:24:36 UTC] "GET /database HTTP/1.1" 200 66450 web_1 | 2018-10-17_09:24:36.53587 - -> /database web_1 | 2018-10-17_09:24:38.80082 ::1 - - [17/Oct/2018:09:24:38 UTC] "GET /process HTTP/1.1" 200 1514 web_1 | 2018-10-17_09:24:38.80084 - -> /process web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== web_1 | [2018-10-17 09:24:41] 127.0.0.1 - - [17/Oct/2018:09:24:41 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/1.8.2" web_1 | web_1 | ==> /var/log/gitlab/nginx/gitlab_access.log <== web_1 | 127.0.0.1 - - [17/Oct/2018:09:24:45 +0000] "GET /help HTTP/1.1" 200 37695 "" "curl/7.59.0" web_1 | web_1 | ==> /var/log/gitlab/gitlab-workhorse/current <== web_1 | 2018-10-17_09:24:45.52837 localhost @ - - [2018/10/17:09:24:45 +0000] "GET /help HTTP/1.1" 200 37643 "" "curl/7.59.0" 0.405 web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production_json.log <== web_1 | {"method":"GET","path":"/help","format":"*/*","controller":"HelpController","action":"index","status":200,"duration":373.98,"view":365.62,"db":4.02,"time":"2018-10-17T09:24:45.153Z","params":[],"remote_ip":"127.0.0.1","user_id":null,"username":null,"ua":"curl/7. 59.0"} web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production.log <== web_1 | Started GET "/help" for 127.0.0.1 at 2018-10-17 09:24:45 +0000 web_1 | Processing by HelpController#index as */* web_1 | Completed 200 OK in 373ms (Views: 365.6ms | ActiveRecord: 4.0ms) web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production_json.log <== web_1 | {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.95,"view":0.27,"db":0.0,"time":"2018-10-17T09:24:45.831Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null} web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production.log <== web_1 | Started GET "/-/metrics" for 127.0.0.1 at 2018-10-17 09:24:45 +0000 web_1 | Processing by MetricsController#index as HTML web_1 | Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms) web_1 | web_1 | ==> /var/log/gitlab/gitlab-monitor/current <== web_1 | 2018-10-17_09:24:49.62888 ::1 - - [17/Oct/2018:09:24:49 UTC] "GET /sidekiq HTTP/1.1" 200 15458 web_1 | 2018-10-17_09:24:49.62892 - -> /sidekiq web_1 | 2018-10-17_09:24:51.53742 ::1 - - [17/Oct/2018:09:24:51 UTC] "GET /database HTTP/1.1" 200 66450 web_1 | 2018-10-17_09:24:51.53753 - -> /database web_1 | 2018-10-17_09:24:53.79972 ::1 - - [17/Oct/2018:09:24:53 UTC] "GET /process HTTP/1.1" 200 1513 web_1 | 2018-10-17_09:24:53.79973 - -> /process web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== web_1 | [2018-10-17 09:24:56] 127.0.0.1 - - [17/Oct/2018:09:24:56 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/1.8.2" web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production_json.log <== web_1 | {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":4.73,"view":0.31,"db":0.0,"time":"2018-10-17T09:25:00.829Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null} web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production.log <== web_1 | Started GET "/-/metrics" for 127.0.0.1 at 2018-10-17 09:25:00 +0000 web_1 | Processing by MetricsController#index as HTML web_1 | Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms) web_1 | web_1 | ==> /var/log/gitlab/gitlab-monitor/current <== web_1 | 2018-10-17_09:25:04.63134 ::1 - - [17/Oct/2018:09:25:04 UTC] "GET /sidekiq HTTP/1.1" 200 15458 web_1 | 2018-10-17_09:25:04.63137 - -> /sidekiq web_1 | 2018-10-17_09:25:06.53905 ::1 - - [17/Oct/2018:09:25:06 UTC] "GET /database HTTP/1.1" 200 66450 web_1 | 2018-10-17_09:25:06.53906 - -> /database web_1 | web_1 | ==> /var/log/gitlab/postgresql/current <== web_1 | 2018-10-17_09:25:08.30652 FATAL: terminating connection due to idle-in-transaction timeout web_1 | web_1 | ==> /var/log/gitlab/gitlab-monitor/current <== web_1 | 2018-10-17_09:25:08.80384 ::1 - - [17/Oct/2018:09:25:08 UTC] "GET /process HTTP/1.1" 200 1513 web_1 | 2018-10-17_09:25:08.80385 - -> /process web_1 | web_1 | ==> /var/log/gitlab/nginx/gitlab_access.log <== web_1 | 192.168.10.128 - - [17/Oct/2018:09:25:09 +0000] "POST /projects HTTP/1.1" 502 2940 "http://gitlabServer/projects/new" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" web_1 | web_1 | ==> /var/log/gitlab/gitlab-workhorse/current <== web_1 | 2018-10-17_09:25:09.03419 time="2018-10-17T09:25:09Z" level=error msg=error correlation-id=XBSF8ExqHM1 error="badgateway: failed after 61s: EOF" method=POST uri=/projects web_1 | 2018-10-17_09:25:09.03465 gitlabServer @ - - [2018/10/17:09:25:09 +0000] "POST /projects HTTP/1.1" 502 2940 "http://gitlabServer/projects/new?namespace_id=7" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" 60.841 web_1 | web_1 | ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== web_1 | E, [2018-10-17T09:25:09.015521 #613] ERROR -- : worker=0 PID:841 timeout (61s > 60s), killing web_1 | E, [2018-10-17T09:25:09.034933 #613] ERROR -- : reaped #<Process::Status: pid 841 SIGKILL (signal 9)> worker=0 web_1 | I, [2018-10-17T09:25:09.122527 #1287] INFO -- : worker=0 readysince the
badgatewayerror message let me to believe its some kind of a network problem, i also tried to push to a new repository from inside the docker container with localhost but with the same result$ sudo docker exec -it gitlab_web_1 /bin/bash root@gitlabServer:/# cd tmp root@gitlabServer:/tmp# mkdir testRepo root@gitlabServer:/tmp# cd testRepo/ root@gitlabServer:/tmp/testRepo# git init Initialized empty Git repository in /tmp/testRepo/.git/ root@gitlabServer:/tmp/testRepo# touch test.md root@gitlabServer:/tmp/testRepo# git remote add origin http://gitlabServer/test/testRepo.git root@gitlabServer:/tmp/testRepo# git add . root@gitlabServer:/tmp/testRepo# git config --global user.email "user@email.com" root@gitlabServer:/tmp/testRepo# git commit -m "test" [master (root-commit) 146db67] test 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 test.md root@gitlabServer:/tmp/testRepo# git push -u origin master Username for 'http://gitlabServer': user Password for 'http://user@gitlabServer': remote: GitLab is not responding fatal: unable to access 'http://gitlabServer/test/testRepo.git/': The requested URL returned error: 502 root@gitlabServer:/tmp/testRepo# curl gitlabServer <html><body>You are being <a href="http://gitlabServer/users/sign_in">redirected</a>.</body></html> root@gitlabServer:/tmp/testRepo# git remote set-url origin http://localhost/test/testRepo.git root@gitlabServer:/tmp/testRepo# git push -u origin master Username for 'http://localhost': user Password for 'http://user@localhost': remote: GitLab is not responding fatal: unable to access 'http://localhost/test/testRepo.git/': The requested URL returned error: 502 root@gitlabServer:/tmp/testRepo# git remote set-url origin http://localhost/test/testRepo.git^C root@gitlabServer:/tmp/testRepo# git push -u origin master Username for 'http://localhost': root Password for 'http://root@localhost': remote: GitLab is not responding fatal: unable to access 'http://localhost/test/testRepo.git/': The requested URL returned error: 502 root@gitlabServer:/tmp/testRepo# exit exitthe 404 error page after i increased the timeout also states
The page could not be found or you don't have permission to view it. The resource that you are attempting to access does not exist or you don't have the necessary permissions to view it. Make sure the address is correct and that the page hasn't moved. Please contact your GitLab administrator if you think this is a mistake.and the resulting log:
web_1 | ==> /var/log/gitlab/nginx/gitlab_access.log <== web_1 | 192.168.10.128 - - [17/Oct/2018:10:22:42 +0000] "POST /projects HTTP/1.1" 302 97 "http://gitlabServer/projects/new" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" web_1 | 192.168.10.128 - - [17/Oct/2018:10:22:42 +0000] "GET /test/blarg HTTP/1.1" 404 3108 "http://gitlabServer/projects/new" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" web_1 | 192.168.10.128 - - [17/Oct/2018:10:22:42 +0000] "GET /favicon.ico HTTP/1.1" 499 0 "" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" web_1 | 192.168.10.128 - - [17/Oct/2018:10:22:42 +0000] "GET /assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png HTTP/1.1" 200 1611 "" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" web_1 | web_1 | ==> /var/log/gitlab/gitlab-workhorse/current <== web_1 | 2018-10-17_10:22:42.21430 gitlabServer @ - - [2018/10/17:10:22:42 +0000] "POST /projects HTTP/1.1" 302 97 "http://gitlabServer/projects/new?namespace_id=7" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" 240.350 web_1 | 2018-10-17_10:22:42.32076 gitlabServer @ - - [2018/10/17:10:22:42 +0000] "GET /test/blarg HTTP/1.1" 404 3108 "http://gitlabServer/projects/new?namespace_id=7" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" 0.103 web_1 | 2018-10-17_10:22:42.64942 time="2018-10-17T10:22:42Z" level=error msg=error correlation-id=7wBxZRyQao5 error="badgateway: failed after 0s: context canceled" method=GET uri=/favicon.ico web_1 | 2018-10-17_10:22:42.64943 time="2018-10-17T10:22:42Z" level=info msg="Send static file" correlation-id=sRAtNgZOQKa encoding= file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png method=GET uri=/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png web_1 | 2018-10-17_10:22:42.64944 gitlabServer @ - - [2018/10/17:10:22:42 +0000] "GET /favicon.ico HTTP/1.1" 502 2940 "" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0" 0.001 web_1 | 2018-10-17_10:22:42.64945 gitlabServer @ - - [2018/10/17:10:22:42 +0000] "GET /assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png HTTP/1.1" 200 1611 "" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62 .0" 0.001 web_1 | web_1 | ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== web_1 | FATAL: terminating connection due to idle-in-transaction timeout web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production_json.log <== web_1 | {"method":"POST","path":"/projects","format":"html","controller":"ProjectsController","action":"create","status":302,"duration":240331.95,"view":0.0,"db":31.54,"location":"http://gitlabServer/test/blarg","time":"2018-10-17T10:18:41.880Z","params":[{"key":"utf8" ,"value":"✓"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"project","value":{"ci_cd_only":"false","name":"blarg","namespace_id":"7","path":"blarg","description":""}}],"remote_ip":"192.168.10.128","user_id":5,"username":"user","ua":"Mozilla/5.0 (Windows NT 6.1 ; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0","gitaly_calls":3} web_1 | {"method":"GET","path":"/test/blarg","format":"html","controller":"ApplicationController","action":"route_not_found","status":404,"duration":79.14,"view":27.14,"db":10.92,"time":"2018-10-17T10:22:42.240Z","params":[{"key":"unmatched_route","value":"test/blarg"}] ,"remote_ip":"192.168.10.128","user_id":5,"username":"user","ua":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0"} web_1 | web_1 | ==> /var/log/gitlab/gitlab-rails/production.log <== web_1 | Unable to save project. Error: PG::ConnectionBad: PQsocket() can't get socket descriptor: ROLLBACK Project ID: 127 web_1 | Redirected to http://gitlabServer/test/blarg web_1 | Completed 302 Found in 240331ms (ActiveRecord: 31.5ms) web_1 | Started GET "/test/blarg" for 192.168.10.128 at 2018-10-17 10:22:42 +0000 web_1 | Processing by ApplicationController#route_not_found as HTML web_1 | Parameters: {"unmatched_route"=>"test/blarg"} web_1 | Completed 404 Not Found in 78ms (Views: 27.1ms | ActiveRecord: 10.9ms)Workaround
I have a somewhat functioning workaround though.
if i create a new project using a template it sometimes works.
Often it creates the project but fails to import the template, rendering it useless since its kinda stuck on the import page.
Sometimes imports successfully and I just force push my local project over it.
And in some rare cases it notices it could not import the template and successfully creates an empty new project.its annoying, but it gets you further when you dont have time to look into the problem
Hi, @miggonaut. Thanks for the clew. Your workaround works for my case too. I hope the developers pay attention on the bug and fix it in the nearest future.
Edited by Aleksey Timin
This looks like a timeout in unicorn. There was another similar report here reporting timeouts displaying Merge Requests:
added Manage [DEPRECATED] bugperformance devopsmanage project labels
added [deprecated] Accepting merge requests label
added groupoptimize label
removed [deprecated] Accepting merge requests label
mentioned in merge request gitlab-org/quality/triage-ops!268 (closed)
Is there a fix for this issue? I'm seeing the same problem. Also upgraded to gitlab-ce-12.2.1-ce.0.el7.x86_64 but the issue remains. Cannot create new projects. I tried increasing the unicorn timeout to 600, but then get a 500 error instead of 502.
Seeing some postgres errors, is this problem a db issue? What can I check to resolve this issue?
=> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-08-25T20:07:02.772Z","class":"BackgroundMigrationWorker","retry":3,"queue":"background_migration","jid":"6cf39c5505f06062f0428e90","created_at":"2019-08-25T20:05:03.574Z","correlation_id":"01343a23-a0cd-4d7a-9f42-7067275623e8","enqueued_at":"2019-08-25T20:07:02.770Z","error_message":"uninitialized constant Gitlab::BackgroundMigration::PopulateMergeRequestMetricsWithEventsDataImproved\nDid you mean? Gitlab::BackgroundMigration::PopulateMergeRequestAssigneesTable","error_class":"NameError","failed_at":1566763594.2144904,"retry_count":0,"pid":6835,"message":"BackgroundMigrationWorker JID-6cf39c5505f06062f0428e90: start","job_status":"start","scheduling_latency_s":0.002} {"severity":"INFO","time":"2019-08-25T20:07:02.791Z","class":"BackgroundMigrationWorker","retry":3,"queue":"background_migration","jid":"6cf39c5505f06062f0428e90","created_at":"2019-08-25T20:05:03.574Z","correlation_id":"01343a23-a0cd-4d7a-9f42-7067275623e8","enqueued_at":"2019-08-25T20:07:02.770Z","error_message":"uninitialized constant Gitlab::BackgroundMigration::PopulateMergeRequestMetricsWithEventsDataImproved\nDid you mean? Gitlab::BackgroundMigration::PopulateMergeRequestAssigneesTable","error_class":"NameError","failed_at":"2019-08-25T20:06:34.214Z","retry_count":0,"pid":6835,"message":"BackgroundMigrationWorker JID-6cf39c5505f06062f0428e90: done: 0.019 sec","job_status":"done","scheduling_latency_s":0.002,"duration":0.019,"completed_at":"2019-08-25T20:07:02.791Z"} ==> /var/log/gitlab/postgresql/current <== 2019-08-25_20:07:12.60606 FATAL: terminating connection due to idle-in-transaction timeout ==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2019-08-25 13:07:14 -0700 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms) ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.47,"view":0.43,"db":0.0,"time":"2019-08-25T20:07:14.673Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"b6b00881-ed24-4689-8b30-557689ee08b8"} ==> /var/log/gitlab/gitlab-rails/production.log <== Unable to save project. Error: PG::UnableToSend: no connection to the server : ROLLBACK Project ID: 137 Completed 500 Internal Server Error in 65516ms (ActiveRecord: 130.9ms) ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server : SELECT "project_mirror_data".* FROM "project_mirror_data" WHERE "project_mirror_data"."project_id" = 137 LIMIT 1): app/services/projects/create_service.rb:156:in `fail' app/services/projects/create_service.rb:69:in `rescue in execute' app/services/projects/create_service.rb:13:in `execute' app/controllers/projects_controller.rb:55:in `create' lib/gitlab/session.rb:11:in `with_session' app/controllers/application_controller.rb:450:in `set_session_storage' lib/gitlab/i18n.rb:55:in `with_locale' lib/gitlab/i18n.rb:61:in `with_user_locale' app/controllers/application_controller.rb:444:in `set_locale' lib/gitlab/middleware/multipart.rb:103:in `call' lib/gitlab/request_profiler/middleware.rb:17:in `call' lib/gitlab/middleware/go.rb:20:in `call' lib/gitlab/etag_caching/middleware.rb:13:in `call' lib/gitlab/middleware/correlation_id.rb:16:in `block in call' lib/gitlab/middleware/correlation_id.rb:15:in `call' lib/gitlab/middleware/read_only/controller.rb:40:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/basic_health_check.rb:25:in `call' lib/gitlab/request_context.rb:26:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call' lib/gitlab/middleware/release_env.rb:12:in `call' Started GET "/favicon.ico" for 123.123.123.123 at 2019-08-25 13:07:17 -0700 ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"POST","path":"/projects","format":"html","controller":"ProjectsController","action":"create","status":500,"error":"ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server\n: SELECT \"project_mirror_data\".* FROM \"project_mirror_data\" WHERE \"project_mirror_data\".\"project_id\" = 137 LIMIT 1","duration":65516.53,"view":0.0,"db":130.92,"time":"2019-08-25T20:07:17.393Z","params":[{"key":"utf8","value":"✓"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"project","value":{"ci_cd_only":"false","name":"test","namespace_id":"1","path":"test","description":"[FILTERED]","visibility_level":"0"}}],"remote_ip":"123.123.123.123","user_id":1,"username":"jason","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36","queue_duration":null,"gitaly_calls":3,"gitaly_duration":10.09,"correlation_id":"IwZuAwwgJa1"} ==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"IwZuAwwgJa1","duration_ms":65618,"host":"gitlab.mydomain.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mydomain.com/projects/new","remote_addr":"123.123.123.123:0","remote_ip":"123.123.123.123","status":500,"system":"http","time":"2019-08-25T13:07:17-07:00","uri":"/projects","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36","written_bytes":2926} {"correlation_id":"SZ7aAtGSXm8","duration_ms":16,"host":"gitlab.mydomain.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mydomain.com/projects","remote_addr":"123.123.123.123:0","remote_ip":"123.123.123.123","status":301,"system":"http","time":"2019-08-25T13:07:17-07:00","uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36","written_bytes":180} {"correlation_id":"ZZPO9gNvHR1","encoding":"","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png","level":"info","method":"GET","msg":"Send static file","time":"2019-08-25T13:07:17-07:00","uri":"/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png"} {"correlation_id":"ZZPO9gNvHR1","duration_ms":0,"host":"gitlab.mydomain.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mydomain.com/projects","remote_addr":"123.123.123.123:0","remote_ip":"123.123.123.123","status":200,"system":"http","time":"2019-08-25T13:07:17-07:00","uri":"/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36","written_bytes":1611} ==> /var/log/gitlab/nginx/gitlab_access.log <== 127.0.0.1 - - [25/Aug/2019:13:07:17 -0700] "POST /projects HTTP/1.0" 500 2926 "https://gitlab.mydomain.com/projects/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" 127.0.0.1 - - [25/Aug/2019:13:07:17 -0700] "GET /favicon.ico HTTP/1.0" 301 180 "https://gitlab.mydomain.com/projects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" 127.0.0.1 - - [25/Aug/2019:13:07:17 -0700] "GET /assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png HTTP/1.0" 200 1611 "https://gitlab.mydomain.com/projects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== FATAL: terminating connection due to idle-in-transaction timeoutEdited by Jason Separovicadded 1 deleted label and removed devopsmanage groupoptimize labels
Setting devopsmanage based on Manage [DEPRECATED].
added devopsmanage label
-
Hey!
GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current
gitlab-cerepository will become a read-only mirror, without any proprietary code. All development is moved to the currentgitlab-eerepository, which we will rename to justgitlabin the coming weeks. As part of this migration, issues will be moved to the currentgitlab-eeproject.If you have any questions about all of this, please ask them in our dedicated FAQ issue.
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855You can also find more information here:
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304- https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/
Frequently Asked Questions
For an up to date list of questions and answers, please take a look at
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855What will we do with the repository names?
https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.
Why rename gitlab-ce to gitlab-foss?
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear
I created a merge requests for CE, and this got closed. What do I need to do?
You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.
How does the licensing work in this new setup?
Everything in the
ee/directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in theee/directory, the process of contributing changes is the same as when using the gitlab-ce repository.Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?
Yes, these changes will be synced to gitlab-foss automatically, several times per day.
Please register or sign in to reply