unable to create new projects in 8.4.3 (fails to create wiki)

I'm using the debian package and it was working till 8.4.0

Unable to save project: Failed to resolve path '/var/lib/gitlab/repositories/root/test.wiki.git': No such file or directory

@marin @jacobvosmaer

I can see test.git is successfully created (so paths, permissions and gitlab-shell is working correctly), but only wiki is failing. I have tried creating more projects all have the same problem.

sudo ls /var/lib/gitlab/repositories/root/
test.git

Added ~152823 label

gitlab:check is failing now. I think it needs a newer gitlab_git.

fatal: Not a git repository (or any of the parent directories): .git 
Checking GitLab Shell ...

GitLab Shell version >= 2.6.10 ? ... OK (2.6.10) 
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by gitlab:gitlab? ... yes
Repo base access is drwxrws---? ... yes
hooks directories in repos are links: ... 
Administrator / test ... rake aborted!
NoMethodError: undefined method `branch_count' for #<Gitlab::Git::Repository:0x00000008710428>
/usr/share/gitlab/app/models/repository.rb:60:in `has_visible_content?'
/usr/share/gitlab/app/models/project.rb:634:in `empty_repo?'
/usr/share/gitlab/lib/tasks/gitlab/check.rake:486:in `block in check_repos_hooks_directory_is_link'
/usr/share/gitlab/lib/tasks/gitlab/check.rake:482:in `check_repos_hooks_directory_is_link'
/usr/share/gitlab/lib/tasks/gitlab/check.rake:366:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:check => gitlab:gitlab_shell:check
(See full trace by running task with --trace)

mentioned in issue #13246 (closed)

Updating gitlab_git to 8.1.0 fixes this error. So Gemfile should be updated to set minimum required version of gitlab_git. It is currently ~> 7.2.22.

/cc: @jacobvosmaer

@pravi gitlab_git 8.0+ is incompatible with GitLab 8.4.3 so whatever was wrong, updating gitlab_git to 8.1.0 is not the solution.

What exact GitLab version are you using?

@jacobvosmaer I will try gitlab_git 7.2.24 then. I'm using gitlab 8.4.3. Also will retry it in a clean chroot environment.

Internal server error is gone after updating gitlab_git to 7.2.24. I'll send a MR to update Gemfile.

But still creating repo is failing.

Unable to save project: Failed to create repository in production.log and Failed to create repository via gitlab-shell in the web ui.

After updating gitlab_git to 7.2.24 and doing a reinstall, creating projects is working again.

@pravi do you want to declare this issue a fluke?

(glad to hear the problem went away)

@jacobvosmaer It was not fluke, Gemile should be updated to require gitlab_git ~> 7.2.24. I will submit a merge request.

@pravi 7.2.24 is already required in Gemfile.lock:

https://gitlab.com/gitlab-org/gitlab-ce/blob/v8.4.3/Gemfile.lock#L359

How are you installing gitlab_git?

Let me rephrase. If you run bundle install on 8.4.3, how could you end up with gitlab_git 7.2.22? That is what I am curious about.

@jacobvosmaer I run 'bundle install --local' and rely on Gemfile only as it is impossible to match Gemfile.lock in debian.

All the dependencies get installed via apt. I have to add all dependencies from Gemfile to their corresponding package names http://anonscm.debian.org/cgit/pkg-ruby-extras/gitlab.git/tree/debian/control

mentioned in merge request !2772 (merged)

it is impossible to match Gemfile.lock in debian.

That flies completely against how GitLab is developed and meant to be deployed.

GitLab has many Gem dependencies that are pinned in Gemfile.lock only. If you do not use exactly those Gem versions specified in Gemfile.lock in your packaged version anything can break in any GitLab release that you package.

@pravi maybe you have discussed the issue of following Gemfile.lock already with @sytses and I am not up-to-date on this topic.

Either way this is a serious challenge for your project. cc @DouweM

We should have a phone call to discuss it I think.

@pravi maybe I misunderstand. Are you saying that you are translating Gemfile.lock into a control file? If we can do that maybe we can make it work (although I am not sure what it would mean for GitLab's chances to coexist with other Ruby programs that share its dependencies if GitLab's control file has super strict version dependencies).

@jacobvosmaer I convert the dependencies from Gemfile and not Gemfile.lock.

You can see a dependency comparison here http://debian.fosscommunity.in/compare/?appname=diaspora&appname=gitlab

Out of 128 dependencies gitlab share with diaspora, 114 match exact version as per Gemfile.

For the remaining, we have to match the requirements either by updating gitlab Gemfile or debian package. See https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name=gem+update

Usually we prefer to move up to newer versions. @rspeicher has been doing the gem updates mostly. There were discussions with @dzaporozhets on this topic, I can't find the exact issue right now.

@pravi thanks for filling me in, I am glad to hear this has been discussed already.

@rspeicher do we have something written down about this somewhere? "How to handle Gem dependencies without making @pravi's life harder"?

@jacobvosmaer I see no other option than to work with @pravi to make the Debian way work. We will update when requested https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name=gem+update

@pravi what is the priority of the https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name=gem+update issues, are some more important than others, is this blocking packaging?

@Sytse currently we use the versions available in debian for those and we fixed all the major issues we found so far.

We hope to see it migrate to testing (next stable) in 5 days if we don't find any release critical bugs by then.

You can the status of the package (version, issues etc) at

https://tracker.debian.org/pkg/gitlab

We have not identified any specific issues that are blocking, but major version difference is more likely to creaye issues, so focusing on them would be nice.

I have already started work on getting this to debian stable.

https://gitlab.pxq.in is running the native debian package (only manual steps were getting ssl certificate, that too automated via letsencrypt command and configuring email, which is already fixed in git).

Note: confirmation emails are delivered to spam at the moment. Adding SPF record might help there, got to check that.

If I really have to make a priority list, then it would be, (but none are blocking at the moment).

1. carrierwave #2805 (closed)
2. unicorn #2820 (closed)
3. sprockets #6069 (closed)
4. http_parser.rb #13333 (closed)

That tells me, we are closer than I thought we are :)

mentioned in commit 4a07cc3c

mentioned in commit b3889b7f

@pravi wow, testing in 4-5 days from now and stable after that would be awesome!

We're pretty busy with 8.5 right now (first rc made) so I don't think that we'll do any updates. But please let me know if something becomes a blocker.

I looked at https://gitlab.pxq.in/ but it gives a 500 error right now. Yesterday I was so excited about all this that I upgraded an ubuntu instance to 16.04 to see if I could install it already, while I managed to upgrade from the command line I didn't set the right package server to find it. But knowing that it is this close I'll just wait for it to land in stable. I can't wait to announce that it landed.

@sytses 3 more rc bugs were reported which I have fixed now. So the clock is reset to 5 days from now (if there are no new rc bugs).

The 500 error issue on https://gitlab.pxq.in should be fixed by now (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=814503) I missed including some files in the package which were added in new releases. I have added a check now so this won't happen for future releases.

You can add deb http://debian.sil.at/debian/ sid main to your sources.list and try apt-get update && apt-get install gitlab once you see 8.4.3+dfsg-4 is available (apt-cache policy gitlab).

Or if you are still impatient, you add one more line deb https://people.debian.org/~praveen/gitlab unreleased main and get the package right now

@pravi I'm impatient and I gave it a go but I got a The following signatures couldn't be verified because the public key is not available: NO_PUBKEY CE1F9C674512C22A. Don't help me, please just carry on with the packaging work. I just said it to show I care

Hoping for no new rc bugs!

@sytses systemd is giving me a hard time. I got to follow FHS and move all files modified at runtime to /var and /run. /run needs extra handling as /run/gitlab should be recreated at every reboot. I got help from debian-devel list. Currently the package is broken when trying to move pids and sockets to /run/gitlab. I will let you know once its fixed. gitlab-workhorse is running, but nginx get 'Permission Denied' error. I have to allow www-data user to access gitlab-workhorse.socket

@sytses I have fixed the issue and uploaded to debian.org main repo. You can remove my people.debian.org/~praveen repo and install when 8.4.3+dfsg-8 is available (may take a few hours to see it after upload).

Making gitlab FHS compliant (move gitlab user's home to /var) is the last rc bug now.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=814476 has details.

Also the current test machine we use (gitlab.pxq.in) is an openvz vm on a wheezy host so it does not have systemd. We are trying to get another test machine with systemd. Currently I have to stop X in my 2GB laptop to get gitlab to run.

@sytses @pravi I just find out that debian package does not use Gemfile.lock. I think we should do something with it. Its absolutely unacceptable to have GitLab running on different dependencies then ANY other GitLab installation. If someone has a problem (bug) with GitLab deb package we would never find out neither we can reproduce it. Its disaster from software development and support perspective.

I see no other option

Make sure gems from our Gemfile.lock are available on Debian by either packing gems or changing our Gemfile.lock.

cc @jacobvosmaer @JobV @marin

@dzaporozhets I understand your concern but unfortunately Debian packaging works different than Bundler packaging. @pravi is doing a great job creating issues to keep GitLab and Discourse in sync so we can use the same version of gems that we share. I see no other solution.

@sytses @pravi I understand why it works this way now and I completely agree @pravi is doing a great job so far.

Still I am wondering if we can improve it in next iteration. I have a next proposal. If we can not make Debian with with our Gemfile.lock can we at least build another Gemfile.lock for Debian only for next purpose:

• it takes no efforts expect once running bundle install on Debian package and saving file somewhere
• it ensures Debian package of GitLab uses same dependencies across different machines
• any developer can download this Gemfile.lock file and reproduce application state on own machine

@pravi what do you think?

Closing due to lack of activity.

Status changed to closed

@dzaporozhets just stating here for the record. We now run about 700 tests with debian's Gemfile.lock so we can find out if there is any issues.

Finished in 8 minutes 7 seconds (files took 13.63 seconds to load)
700 examples, 0 failures

We can run only this much because gitlab tests are expecting to be run only with RAILS_ENV=test and we run tests with RAILS_ENV=production (a fully installed package). I'm planning to make tests run with RAILS_ENV=test in debian so we can more confidently support debian's Gemfile.lock. See https://gitlab.com/gitlab-org/gitlab-ce/issues/22291 for a background.

Mentioned in issue #19262 (closed)

@pravi thanks for update . Looking forward to see whole test suite running on modified GitLab package for Debian

@dzaporozhets we are at 4881 tests now we are increasing the test coverage gradually. We are skipping some failing tests for now, some of them could be missing configuration, others would need more more investigation.

https://ci.debian.net/data/packages/unstable/amd64/g/gitlab/latest-autopkgtest/log.gz

Finished in 36 minutes 41 seconds (files took 11.35 seconds to load)
4881 examples, 0 failures, 7 pending 

@pravi great news. Do you know percentage test coverage for current state?