502 GitLab is not responding v7.9.4

Moved from: omnibus-gitlab#678 (closed)

tl;dr: huge commits caused a 502 on our tiny server when reviewing commit history. Updating from 7.9.4 to 7.13 showed us, what I believe to be, the source of our problems.

Thank you to you developers who create these beautiful error messages. You've saved my head from a lot of hurt.

Edit: omitted personal details and added log files.

Description:

I haven't seen a solution for my exact problem so I thought I'd post it here. We have an issue with GitLab where users either see a "502 GitLab is not responding" or "502 Bad Gateway" when attempting to browse the history of certain commits. We identified that the issue stemm from one particular user who reported no unusual activity. Strangely enough, when I attempt to open those commits I would see the CPU spike to 100% on a core, my web browser (well technically I think it's GitLab) take a minute to time out, and this would possibly stop me from loading other pages (only had one core allocated at the time) until the operation timed out.

Since updating (on a clone, on an isolated network), I've started seeing the attached screenshots error message across, what I think are, all the troublesome commits. I apologize for stating the obvious but I believe that our error is exactly as the 7.13 error message state's, that there are too many changes. What do you guys thing? I'm an intern so my skill/expertise is close to zilch.

Here are some log snippits (sensitive information modified):

gitlab-rake gitlab:env:info

System information
System:		Ubuntu 14.04
Current User:	git
Using RVM:	no
Ruby Version:	2.1.5p273
Gem Version:	2.2.1
Bundler Version:1.5.3
Rake Version:	10.4.2
Sidekiq Version:3.3.0

GitLab information
Version:	7.9.4
Revision:	b10de29
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		http://gitlab.example.net
HTTP Clone URL:	http://gitlab.example.net/some-project.git
SSH Clone URL:	git@gitlab.example.com:some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	2.6.2
Repositories:	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

7.9.4 Error Message

/var/log/gitlab/nginx/gitlab_error.log

2015/07/22 19:03:09 [error] 1676#0: *1 connect() to 
unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: 
Connection refused) while connecting to upstream, client: 192.168.1.3, 
server: gitlab.example.com, request: "GET / HTTP/1.1", upstream: 
"http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/", 
host: "gitlab"
2015/07/22 19:04:24 [error] 1676#0: *1 upstream prematurely closed 
connection while reading response header from upstream, client: 
192.168.1.3, server: gitlab.example.com, request: "GET 
/group5/redactedproject/commit/aaaaabbbbbbccccc12345 HTTP/1.1", 
upstream: 
"http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/group5/online-path-scheduling/commit/aaaaabbbbbbccccc12345",
 host: "gitlab", referrer: "http://gitlab/groups/group5"

/var/log/gitlab/unicorn/unicorn_stderr.log

W, [2015-07-22T12:54:00.449164 #12371]  WARN -- : 
#<Unicorn::HttpServer:0x0000000182a590>: worker (pid: 12371) 
exceeds memory limit (237359104 bytes > 235555461 bytes)
W, [2015-07-22T12:54:00.449243 #12371]  WARN -- : Unicorn::WorkerKiller 
send SIGQUIT (pid: 12371) alive: 959 sec (trial 1)
I, [2015-07-22T12:54:00.663417 #1170]  INFO -- : reaped 
#<Process::Status: pid 12371 exit 0> worker=1
I, [2015-07-22T12:54:00.665650 #14266]  INFO -- : worker=1 spawned 
pid=14266
I, [2015-07-22T12:54:00.665903 #14266]  INFO -- : worker=1 ready

/var/log/gitlab/redis/

2015-07-22_22:58:45.53032 [1863] 22 Jul 18:58:45.529 # WARNING 
overcommit_memory is set to 0! Background save may fail under low memory
 condition. To fix this issue add 'vm.overcommit_memory = 1' to 
/etc/sysctl.conf and then reboot or run the command 'sysctl 
vm.overcommit_memory=1' for this to take effect.

I tried increasing the RAM and number of cores allocated. Attempting to increase the number of unicorn workers and some other suggested fixes. It wasn't until I updated following the guide below to v9.13 that I saw what was REALLY going on.

Saw new message after following this update guide

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/upgrader.md

7.13 Error Message