Sidekiq creeping mem usage
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=20797)
</details>
<!--IssueSummary end-->
### Summary
Since an upgrade to Gitlab 10.3.5 from 9.2.2 I have been noticing abnormal memory usage by sidekiq. I have experienced a kernel panic, but unfortunately wasn't actively monitoring at the time, and so cant say they are directly related. A slight twist in the plot is that I also performed a patch (for meltdown) on Debian Jessie.
### Steps to reproduce
Below is a link to my docker compose config which is exhibiting the problem. The only difference is passwords and possibly Redis version.
https://gitlab.com/snippets/1695148
To start on a machine with Docker and compose; `$ docker-compose up -d`
One point of note is my version of Redis hasn't updated recently.
```
$ docker exec -it docker_redis_1 redis-server --version
Redis server v=2.8.4 sha=00000000:0 malloc=jemalloc-3.4.1 bits=64 build=a44a05d76f06a5d9
```
This may not actually be `latest` in Dockerhub.
### What is the current *bug* behavior?
The allocated memory for sidekiq is increasing over time, below is a list of the largest process memory and process path:
```
$ ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n
77.9453 MB postgres:
78.6445 MB postgres:
79.6289 MB postgres:
84.2578 MB postgres:
85.5039 MB postgres:
89.7578 MB postgres:
134.75 MB ruby
134.836 MB ruby
209.062 MB /opt/conda/bin/python
323.52 MB unicorn_rails
420.031 MB unicorn_rails
445.559 MB unicorn_rails
446.902 MB unicorn_rails
845.703 MB sidekiq
```
Whilst [this](https://gitlab.com/gitlab-org/gitlab-ce/blob/619b350279c6e3a987cefce935fdaf7ab13632ff/doc/install/requirements.md#redis-and-sidekiq) would suggest a growing memory is normal, 800mb for 27 users (probably only 3 active at any given time) feels extreme.
I am running Debian Jessie `3.16.0-5-amd64 #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) x86_64 GNU/Linux`.
### What is the expected *correct* behavior?
Whilst sidekiq is idle, I wouldn't expect to see memory consumption increases. The peak memory usage being recorded in the dashboard is `16.24M`.
### Relevant logs and/or screenshots
### Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
#### Results of GitLab environment info
`docker-compose exec --user git gitlab bundle exec rake gitlab:env:info RAILS_ENV=production`
```
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.3.5p376
Gem Version: 2.5.2.1
Bundler Version:1.16.1
Rake Version: 12.3.0
Redis Version: 2.8.4
Git Version: 2.15.1
Sidekiq Version:5.0.4
Go Version: unknown
GitLab information
Version: 10.3.5
Revision: 7c30d48
Directory: /home/git/gitlab
DB Adapter: postgresql
URL: http://myhost:10080
HTTP Clone URL: http://myhost:10080/some-group/some-project.git
SSH Clone URL: ssh://git@myhost:10022/some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 5.10.2
Repository storage paths:
- default: /home/git/data/repositories
Hooks: /home/git/gitlab-shell/hooks
Git: /usr/bin/git
```
#### Results of GitLab application Check
`$ docker-compose exec --user git gitlab bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`
```
Checking GitLab Shell ...
GitLab Shell version >= 5.10.2 ? ... OK (5.10.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/4 ... ok
6/9 ... ok
6/11 ... ok
6/12 ... ok
4/13 ... ok
5/14 ... repository is empty
8/15 ... ok
12/16 ... ok
14/17 ... ok
7/18 ... ok
9/19 ... ok
5/22 ... ok
8/23 ... ok
8/24 ... ok
4/26 ... ok
4/27 ... ok
4/28 ... ok
4/31 ... ok
4/32 ... ok
2/33 ... ok
6/34 ... ok
6/35 ... ok
4/36 ... ok
2/42 ... ok
18/43 ... ok
9/44 ... ok
19/45 ... ok
20/46 ... ok
14/47 ... ok
4/48 ... ok
2/49 ... ok
4/50 ... ok
5/52 ... ok
2/53 ... ok
8/55 ... ok
4/56 ... ok
19/57 ... ok
20/58 ... ok
2/59 ... ok
27/60 ... ok
28/61 ... ok
31/62 ... ok
29/63 ... ok
4/64 ... ok
31/65 ... ok
30/67 ... ok
30/68 ... ok
30/69 ... ok
31/70 ... ok
5/71 ... ok
5/72 ... ok
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /home/git/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml
Checking LDAP ...
Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
<< Ommitted >>
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /home/git/data/uploads
sudo find /home/git/data/uploads -type f -exec chmod 0644 {} \;
sudo find /home/git/data/uploads -type d -not -path /home/git/data/uploads -exec chmod 0700 {} \;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... yes
Init script up-to-date? ... yes
Projects have namespace: ...
4/4 ... yes
6/9 ... yes
6/11 ... yes
6/12 ... yes
4/13 ... yes
5/14 ... yes
8/15 ... yes
12/16 ... yes
14/17 ... yes
7/18 ... yes
9/19 ... yes
5/22 ... yes
8/23 ... yes
8/24 ... yes
4/26 ... yes
4/27 ... yes
4/28 ... yes
4/31 ... yes
4/32 ... yes
2/33 ... yes
6/34 ... yes
6/35 ... yes
4/36 ... yes
2/42 ... yes
18/43 ... yes
9/44 ... yes
19/45 ... yes
20/46 ... yes
14/47 ... yes
4/48 ... yes
2/49 ... yes
4/50 ... yes
5/52 ... yes
2/53 ... yes
8/55 ... yes
4/56 ... yes
19/57 ... yes
20/58 ... yes
2/59 ... yes
27/60 ... yes
28/61 ... yes
31/62 ... yes
29/63 ... yes
4/64 ... yes
31/65 ... yes
30/67 ... yes
30/68 ... yes
30/69 ... yes
31/70 ... yes
5/71 ... yes
5/72 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.15.1)
Git user has default SSH configuration? ... yes
Active users: ... 23
Checking GitLab ... Finished
```
### Possible fixes
As the memory is increasing constantly (with no signs of stopping), we may need to configure Monit to reboot at regular intervals, but this is not ideal.
issue