Sidekiq stopping and blocked PG queries on user removal
ZD: https://gitlab.zendesk.com/agent/tickets/21236
Description
After starting to move repos and users over to production Sidekiq stopped responding and no background jobs where getting processed. Restarting Sidekiq alone timed out. Restarting Postgres does allow Sidekiq restart and Jobs continue processing.
We found that the jobs that kept coming up where related to removing a user from the application, no specific user by the way. Looking through PG blocked queries we found that the blocked queries where related to user removal. Trying to remove the user from the rails console doesn't work either, no output.
Related
Problem related to http://askubuntu.com/questions/299689/tail-inotify-cannot-be-used-reverting-to-polling-too-many-open-files
Env Info
- GitLab 8.9.5
- Output of
rpm -q gitlab: package gitlab is not installed.
GitLab env info
System information
System: RedHatEnterpriseWorkstation 6.5
Current User: git
Using RVM: no
Ruby Version: 2.1.8p440
Gem Version: 2.5.1
Bundler Version:1.10.6
Rake Version: 10.5.0
Sidekiq Version:4.1.2
GitLab information
Version: 8.9.5-ee
Revision: 42dceaa
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.2.17
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers: github
GitLab Shell
Version: 3.0.0
Repositories: /remote/sigsrc/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/gitServer Limits
-bash-4.1$ sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 128
-bash-4.1$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 524288
-bash-4.1$ cat /proc/sys/fs/file-max
1000000
-bash-4.1$ lsof | wc -l
1295
sh-4.1$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 514772
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 5120
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
```
## Logs
### Sidekiq status and GDB
```
sudo gitlab-ctl status
run: sidekiq: (pid 19973) 42153s, got TERM; run: log: (pid 1224) 7656343s
git 19973 0.5 1.0 3641888 718668 ? Ssl Jul12 4:11 sidekiq 4.1.2 gitlab-rails [6 of 25 busy]
gdb –p 19973
NOTE: Missing separate debuginfos, use: debuginfo-install gitlab-ee-8.9.5-ee.0.el6.x86_64
(gdb) info threads
7 Thread 0x7f98761ec700 (LWP 19975) 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
6 Thread 0x7f9869c54700 (LWP 20553) 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
5 Thread 0x7f9869850700 (LWP 20557) 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
4 Thread 0x7f9868f47700 (LWP 20566) 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
3 Thread 0x7f9868e46700 (LWP 20567) 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
2 Thread 0x7f9868a42700 (LWP 20571) 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
* 1 Thread 0x7f9876185700 (LWP 19973) 0x00007f987591a5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
set pagination off
thread apply all bt
Thread 7 (Thread 0x7f98761ec700 (LWP 19975)):
#0 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
#1 0x00007f9875cec359 in timer_thread_sleep (p=0x238e018) at thread_pthread.c:1420
#2 thread_timer (p=0x238e018) at thread_pthread.c:1497
#3 0x00007f98759169d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f9874d9c8fd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f9869c54700 (LWP 20553)):
#0 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
#1 0x00007f986b8ae35f in pqSocketCheck () from /opt/gitlab/embedded/lib/libpq.so.5
#2 0x00007f986b8ae3d0 in pqWaitTimed () from /opt/gitlab/embedded/lib/libpq.so.5
#3 0x00007f986b8adb49 in PQgetResult () from /opt/gitlab/embedded/lib/libpq.so.5
#4 0x00007f986b8addc8 in PQexecFinish () from /opt/gitlab/embedded/lib/libpq.so.5
#5 0x00007f986badd759 in gvl_PQexecPrepared_skeleton (data=<value optimized out>) at gvl_wrappers.c:9
#6 0x00007f9875cf14a2 in call_without_gvl (func=0x7f986badd730 <gvl_PQexecPrepared_skeleton>, data1=0x7f9869c4f240, ubf=<value optimized out>, data2=<value optimized out>, fail_if_interrupted=0) at thread.c:1270
#7 0x00007f986badd2a9 in gvl_PQexecPrepared (conn=<value optimized out>, stmtName=<value optimized out>, nParams=<value optimized out>, paramValues=<value optimized out>, paramLengths=<value optimized out>, paramFormats=<value optimized out>, resultFormat=0) at gvl_wrappers.c:10
#8 0x00007f986bae471c in pgconn_exec_prepared (argc=<value optimized out>, argv=<value optimized out>, self=111377120) at pg_connection.c:1409
#9 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bbeae0, ci=<value optimized out>) at vm_insnhelper.c:1510
#10 0x00007f9875cd0d32 in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:1025
#11 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#12 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=137687320, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=108929120) at vm.c:817
#13 0x00007f9875cdbc63 in vm_invoke_proc (th=0x13816fd0, proc=0x7f9844554210, self=137687320, defined_class=108929120, argc=0, argv=0x14ac0e48, blockptr=0x0) at vm.c:881
#14 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#15 0x00007f9875bac62c in proc_call (argc=0, argv=0x14ac0e48, procval=137682000) at proc.c:724
#16 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bbefe0, ci=<value optimized out>) at vm_insnhelper.c:1510
#17 0x00007f9875cd0d32 in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:1025
#18 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#19 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=139879200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74521920) at vm.c:817
#20 0x00007f9875cdbc63 in vm_invoke_proc (th=0x13816fd0, proc=0x7f98342d9fd0, self=139879200, defined_class=74521920, argc=1, argv=0x14ac0de0, blockptr=0x0) at vm.c:881
#21 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#22 0x00007f9875bac62c in proc_call (argc=1, argv=0x14ac0de0, procval=127615440) at proc.c:724
#23 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bbf0d0, ci=<value optimized out>) at vm_insnhelper.c:1510
#24 0x00007f9875cd131e in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:996
#25 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#26 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=142495120, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x7f575d8, defined_class=109825840) at vm.c:817
#27 0x00007f9875cdc1b6 in vm_yield_with_cref (under=<value optimized out>, self=142495120, values=133527040) at vm.c:848
#28 yield_under (under=<value optimized out>, self=142495120, values=133527040) at vm_eval.c:1529
#29 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bbf530, ci=<value optimized out>) at vm_insnhelper.c:1510
#30 0x00007f9875cdd373 in vm_call_cfunc (th=0x13816fd0, cfp=0x14bbf530, ci=0x46f4250) at vm_insnhelper.c:1600
#31 vm_call_method (th=0x13816fd0, cfp=0x14bbf530, ci=0x46f4250) at vm_insnhelper.c:1788
#32 0x00007f9875cd131e in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:996
#33 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#34 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=69209760, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522600) at vm.c:817
#35 0x00007f9875cdbc63 in vm_invoke_proc (th=0x13816fd0, proc=0x7f9834242990, self=69209760, defined_class=74522600, argc=2, argv=0x14ac0be8, blockptr=0x0) at vm.c:881
#36 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#37 0x00007f9875bac62c in proc_call (argc=2, argv=0x14ac0be8, procval=127000200) at proc.c:724
#38 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bbf5d0, ci=<value optimized out>) at vm_insnhelper.c:1510
#39 0x00007f9875cd0d32 in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:1025
#40 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#41 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=74523200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74523160) at vm.c:817
#42 0x00007f9875cdbc63 in vm_invoke_proc (th=0x13816fd0, proc=0x7f9834791ae0, self=74523200, defined_class=74523160, argc=1, argv=0x14ac0b90, blockptr=0x0) at vm.c:881
#43 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#44 0x00007f9875bac62c in proc_call (argc=1, argv=0x14ac0b90, procval=126999680) at proc.c:724
#45 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bbf670, ci=<value optimized out>) at vm_insnhelper.c:1510
#46 0x00007f9875cd131e in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:996
#47 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#48 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=126770280, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522080) at vm.c:817
#49 0x00007f9875cdc58d in vm_yield (val=126999680) at vm.c:856
#50 rb_yield_0 (val=126999680) at vm_eval.c:944
#51 rb_yield (val=126999680) at vm_eval.c:954
#52 0x00007f9875b592c7 in rb_ary_each (array=<value optimized out>) at array.c:1796
#53 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bbf710, ci=<value optimized out>) at vm_insnhelper.c:1510
#54 0x00007f9875cd131e in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:996
#55 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#56 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=176433920, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=57770080) at vm.c:817
#57 0x00007f9875cdbc63 in vm_invoke_proc (th=0x13816fd0, proc=0x7f98442f3a50, self=176433920, defined_class=57770080, argc=0, argv=0x14ac06d8, blockptr=0x0) at vm.c:881
#58 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#59 0x00007f9875bac62c in proc_call (argc=0, argv=0x14ac06d8, procval=142105680) at proc.c:724
#60 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x13816fd0, reg_cfp=0x14bc0250, ci=<value optimized out>) at vm_insnhelper.c:1510
#61 0x00007f9875cd0d32 in vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:1025
#62 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#63 0x00007f9875cd7089 in vm_call0_body (th=0x13816fd0, ci=0x7f9869c533f0, argv=<value optimized out>) at vm_eval.c:171
#64 0x00007f9875cd7597 in vm_call0 (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:50
#65 rb_vm_call (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:247
#66 0x00007f9875ba94d8 in rb_method_call_with_block (argc=0, argv=0x7a92170, method=41452040, pass_procval=8) at proc.c:1831
#67 0x00007f9875baa93a in bmcall (args=128524640, method=41452040, argc=0, argv=<value optimized out>, passed_proc=8) at proc.c:2306
#68 0x00007f9875ccc3e5 in vm_yield_with_cfunc (th=0x13816fd0, block=0x11ac40b0, self=<value optimized out>, argc=0, argv=0x14ac05b8, blockargptr=<value optimized out>) at vm_insnhelper.c:2136
#69 0x00007f9875cd4846 in vm_invoke_block (th=0x13816fd0, initial=<value optimized out>) at vm_insnhelper.c:2386
#70 vm_exec_core (th=0x13816fd0, initial=<value optimized out>) at insns.def:1064
#71 0x00007f9875cd656f in vm_exec (th=0x13816fd0) at vm.c:1427
#72 0x00007f9875cdb914 in invoke_block_from_c (th=0x13816fd0, block=<value optimized out>, self=130866920, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=103117240) at vm.c:817
#73 0x00007f9875cdbc63 in vm_invoke_proc (th=0x13816fd0, proc=0xe546f70, self=130866920, defined_class=103117240, argc=0, argv=0x27a34a0, blockptr=0x0) at vm.c:881
#74 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#75 0x00007f9875ced51e in thread_start_func_2 (th=0x13816fd0, stack_start=0x7f9869c55000) at thread.c:535
#76 0x00007f9875cf3410 in thread_start_func_1 (th_ptr=0x13816fd0) at thread_pthread.c:846
#77 0x00007f98759169d1 in start_thread () from /lib64/libpthread.so.0
#78 0x00007f9874d9c8fd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f9869850700 (LWP 20557)):
#0 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
#1 0x00007f986b8ae35f in pqSocketCheck () from /opt/gitlab/embedded/lib/libpq.so.5
#2 0x00007f986b8ae3d0 in pqWaitTimed () from /opt/gitlab/embedded/lib/libpq.so.5
#3 0x00007f986b8adb49 in PQgetResult () from /opt/gitlab/embedded/lib/libpq.so.5
#4 0x00007f986b8addc8 in PQexecFinish () from /opt/gitlab/embedded/lib/libpq.so.5
#5 0x00007f986badd759 in gvl_PQexecPrepared_skeleton (data=<value optimized out>) at gvl_wrappers.c:9
#6 0x00007f9875cf14a2 in call_without_gvl (func=0x7f986badd730 <gvl_PQexecPrepared_skeleton>, data1=0x7f986984b240, ubf=<value optimized out>, data2=<value optimized out>, fail_if_interrupted=0) at thread.c:1270
#7 0x00007f986badd2a9 in gvl_PQexecPrepared (conn=<value optimized out>, stmtName=<value optimized out>, nParams=<value optimized out>, paramValues=<value optimized out>, paramLengths=<value optimized out>, paramFormats=<value optimized out>, resultFormat=0) at gvl_wrappers.c:10
#8 0x00007f986bae471c in pgconn_exec_prepared (argc=<value optimized out>, argv=<value optimized out>, self=284260960) at pg_connection.c:1409
#9 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fbeb20, ci=<value optimized out>) at vm_insnhelper.c:1510
#10 0x00007f9875cd0d32 in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:1025
#11 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#12 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=200094520, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=108929120) at vm.c:817
#13 0x00007f9875cdbc63 in vm_invoke_proc (th=0xd425d90, proc=0x7f9840b61ec0, self=200094520, defined_class=108929120, argc=0, argv=0x14ec0e88, blockptr=0x0) at vm.c:881
#14 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#15 0x00007f9875bac62c in proc_call (argc=0, argv=0x14ec0e88, procval=200087080) at proc.c:724
#16 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fbf020, ci=<value optimized out>) at vm_insnhelper.c:1510
#17 0x00007f9875cd0d32 in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:1025
#18 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#19 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=139879200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74521920) at vm.c:817
#20 0x00007f9875cdbc63 in vm_invoke_proc (th=0xd425d90, proc=0x7f98342d9fd0, self=139879200, defined_class=74521920, argc=1, argv=0x14ec0e20, blockptr=0x0) at vm.c:881
#21 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#22 0x00007f9875bac62c in proc_call (argc=1, argv=0x14ec0e20, procval=127615440) at proc.c:724
#23 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fbf110, ci=<value optimized out>) at vm_insnhelper.c:1510
#24 0x00007f9875cd131e in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:996
#25 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#26 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=140290413989280, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x8ecdda0, defined_class=109825840) at vm.c:817
#27 0x00007f9875cdc1b6 in vm_yield_with_cref (under=<value optimized out>, self=140290413989280, values=149741040) at vm.c:848
#28 yield_under (under=<value optimized out>, self=140290413989280, values=149741040) at vm_eval.c:1529
#29 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fbf570, ci=<value optimized out>) at vm_insnhelper.c:1510
#30 0x00007f9875cdd373 in vm_call_cfunc (th=0xd425d90, cfp=0x14fbf570, ci=0x46f4250) at vm_insnhelper.c:1600
#31 vm_call_method (th=0xd425d90, cfp=0x14fbf570, ci=0x46f4250) at vm_insnhelper.c:1788
#32 0x00007f9875cd131e in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:996
#33 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#34 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=69209760, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522600) at vm.c:817
#35 0x00007f9875cdbc63 in vm_invoke_proc (th=0xd425d90, proc=0x7f9834242990, self=69209760, defined_class=74522600, argc=2, argv=0x14ec0c28, blockptr=0x0) at vm.c:881
#36 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#37 0x00007f9875bac62c in proc_call (argc=2, argv=0x14ec0c28, procval=127000200) at proc.c:724
#38 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fbf610, ci=<value optimized out>) at vm_insnhelper.c:1510
#39 0x00007f9875cd0d32 in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:1025
#40 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#41 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=74523200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74523160) at vm.c:817
#42 0x00007f9875cdbc63 in vm_invoke_proc (th=0xd425d90, proc=0x7f9834791ae0, self=74523200, defined_class=74523160, argc=1, argv=0x14ec0bd0, blockptr=0x0) at vm.c:881
#43 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#44 0x00007f9875bac62c in proc_call (argc=1, argv=0x14ec0bd0, procval=126999680) at proc.c:724
#45 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fbf6b0, ci=<value optimized out>) at vm_insnhelper.c:1510
#46 0x00007f9875cd131e in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:996
#47 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#48 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=126770280, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522080) at vm.c:817
#49 0x00007f9875cdc58d in vm_yield (val=126999680) at vm.c:856
#50 rb_yield_0 (val=126999680) at vm_eval.c:944
#51 rb_yield (val=126999680) at vm_eval.c:954
#52 0x00007f9875b592c7 in rb_ary_each (array=<value optimized out>) at array.c:1796
#53 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fbf750, ci=<value optimized out>) at vm_insnhelper.c:1510
#54 0x00007f9875cd131e in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:996
#55 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#56 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=176433920, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=57770080) at vm.c:817
#57 0x00007f9875cdbc63 in vm_invoke_proc (th=0xd425d90, proc=0x7f9840a7a240, self=176433920, defined_class=57770080, argc=0, argv=0x14ec0718, blockptr=0x0) at vm.c:881
#58 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#59 0x00007f9875bac62c in proc_call (argc=0, argv=0x14ec0718, procval=140290411854960) at proc.c:724
#60 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0xd425d90, reg_cfp=0x14fc0290, ci=<value optimized out>) at vm_insnhelper.c:1510
#61 0x00007f9875cd0d32 in vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:1025
#62 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#63 0x00007f9875cd7089 in vm_call0_body (th=0xd425d90, ci=0x7f986984f3f0, argv=<value optimized out>) at vm_eval.c:171
#64 0x00007f9875cd7597 in vm_call0 (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:50
#65 rb_vm_call (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:247
#66 0x00007f9875ba94d8 in rb_method_call_with_block (argc=0, argv=0x3eebf40, method=41787000, pass_procval=8) at proc.c:1831
#67 0x00007f9875baa93a in bmcall (args=65978160, method=41787000, argc=0, argv=<value optimized out>, passed_proc=8) at proc.c:2306
#68 0x00007f9875ccc3e5 in vm_yield_with_cfunc (th=0xd425d90, block=0x13480d20, self=<value optimized out>, argc=0, argv=0x14ec05f8, blockargptr=<value optimized out>) at vm_insnhelper.c:2136
#69 0x00007f9875cd4846 in vm_invoke_block (th=0xd425d90, initial=<value optimized out>) at vm_insnhelper.c:2386
#70 vm_exec_core (th=0xd425d90, initial=<value optimized out>) at insns.def:1064
#71 0x00007f9875cd656f in vm_exec (th=0xd425d90) at vm.c:1427
#72 0x00007f9875cdb914 in invoke_block_from_c (th=0xd425d90, block=<value optimized out>, self=130862600, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=103117240) at vm.c:817
#73 0x00007f9875cdbc63 in vm_invoke_proc (th=0xd425d90, proc=0xed21610, self=130862600, defined_class=103117240, argc=0, argv=0x27e1c50, blockptr=0x0) at vm.c:881
#74 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#75 0x00007f9875ced51e in thread_start_func_2 (th=0xd425d90, stack_start=0x7f9869851000) at thread.c:535
#76 0x00007f9875cf3410 in thread_start_func_1 (th_ptr=0xd425d90) at thread_pthread.c:846
#77 0x00007f98759169d1 in start_thread () from /lib64/libpthread.so.0
#78 0x00007f9874d9c8fd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f9868f47700 (LWP 20566)):
#0 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
#1 0x00007f986b8ae35f in pqSocketCheck () from /opt/gitlab/embedded/lib/libpq.so.5
#2 0x00007f986b8ae3d0 in pqWaitTimed () from /opt/gitlab/embedded/lib/libpq.so.5
#3 0x00007f986b8adb49 in PQgetResult () from /opt/gitlab/embedded/lib/libpq.so.5
#4 0x00007f986b8addc8 in PQexecFinish () from /opt/gitlab/embedded/lib/libpq.so.5
#5 0x00007f986badd759 in gvl_PQexecPrepared_skeleton (data=<value optimized out>) at gvl_wrappers.c:9
#6 0x00007f9875cf14a2 in call_without_gvl (func=0x7f986badd730 <gvl_PQexecPrepared_skeleton>, data1=0x7f9868f42240, ubf=<value optimized out>, data2=<value optimized out>, fail_if_interrupted=0) at thread.c:1270
#7 0x00007f986badd2a9 in gvl_PQexecPrepared (conn=<value optimized out>, stmtName=<value optimized out>, nParams=<value optimized out>, paramValues=<value optimized out>, paramLengths=<value optimized out>, paramFormats=<value optimized out>, resultFormat=0) at gvl_wrappers.c:10
#8 0x00007f986bae471c in pgconn_exec_prepared (argc=<value optimized out>, argv=<value optimized out>, self=142243040) at pg_connection.c:1409
#9 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158bebb0, ci=<value optimized out>) at vm_insnhelper.c:1510
#10 0x00007f9875cd0d32 in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:1025
#11 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#12 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=127619160, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=108929120) at vm.c:817
#13 0x00007f9875cdbc63 in vm_invoke_proc (th=0x133927d0, proc=0x7f98342c3ec0, self=127619160, defined_class=108929120, argc=0, argv=0x157c0f18, blockptr=0x0) at vm.c:881
#14 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#15 0x00007f9875bac62c in proc_call (argc=0, argv=0x157c0f18, procval=127615680) at proc.c:724
#16 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158bf0b0, ci=<value optimized out>) at vm_insnhelper.c:1510
#17 0x00007f9875cd0d32 in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:1025
#18 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#19 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=139879200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74521920) at vm.c:817
#20 0x00007f9875cdbc63 in vm_invoke_proc (th=0x133927d0, proc=0x7f98342d9fd0, self=139879200, defined_class=74521920, argc=1, argv=0x157c0eb0, blockptr=0x0) at vm.c:881
#21 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#22 0x00007f9875bac62c in proc_call (argc=1, argv=0x157c0eb0, procval=127615440) at proc.c:724
#23 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158bf1a0, ci=<value optimized out>) at vm_insnhelper.c:1510
#24 0x00007f9875cd131e in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:996
#25 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#26 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=136370440, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x791d370, defined_class=109825840) at vm.c:817
#27 0x00007f9875cdc1b6 in vm_yield_with_cref (under=<value optimized out>, self=136370440, values=126997400) at vm.c:848
#28 yield_under (under=<value optimized out>, self=136370440, values=126997400) at vm_eval.c:1529
#29 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158bf600, ci=<value optimized out>) at vm_insnhelper.c:1510
#30 0x00007f9875cdd373 in vm_call_cfunc (th=0x133927d0, cfp=0x158bf600, ci=0x46f4250) at vm_insnhelper.c:1600
#31 vm_call_method (th=0x133927d0, cfp=0x158bf600, ci=0x46f4250) at vm_insnhelper.c:1788
#32 0x00007f9875cd131e in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:996
#33 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#34 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=69209760, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522600) at vm.c:817
#35 0x00007f9875cdbc63 in vm_invoke_proc (th=0x133927d0, proc=0x7f9834242990, self=69209760, defined_class=74522600, argc=2, argv=0x157c0cb8, blockptr=0x0) at vm.c:881
#36 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#37 0x00007f9875bac62c in proc_call (argc=2, argv=0x157c0cb8, procval=127000200) at proc.c:724
#38 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158bf6a0, ci=<value optimized out>) at vm_insnhelper.c:1510
#39 0x00007f9875cd0d32 in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:1025
#40 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#41 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=74523200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74523160) at vm.c:817
#42 0x00007f9875cdbc63 in vm_invoke_proc (th=0x133927d0, proc=0x7f9834791ae0, self=74523200, defined_class=74523160, argc=1, argv=0x157c0c60, blockptr=0x0) at vm.c:881
#43 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#44 0x00007f9875bac62c in proc_call (argc=1, argv=0x157c0c60, procval=126999680) at proc.c:724
#45 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158bf740, ci=<value optimized out>) at vm_insnhelper.c:1510
#46 0x00007f9875cd131e in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:996
#47 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#48 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=126770280, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522080) at vm.c:817
#49 0x00007f9875cdc58d in vm_yield (val=126999680) at vm.c:856
#50 rb_yield_0 (val=126999680) at vm_eval.c:944
#51 rb_yield (val=126999680) at vm_eval.c:954
#52 0x00007f9875b592c7 in rb_ary_each (array=<value optimized out>) at array.c:1796
#53 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158bf7e0, ci=<value optimized out>) at vm_insnhelper.c:1510
#54 0x00007f9875cd131e in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:996
#55 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#56 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=176433920, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=57770080) at vm.c:817
#57 0x00007f9875cdbc63 in vm_invoke_proc (th=0x133927d0, proc=0x7f98346ad5e0, self=176433920, defined_class=57770080, argc=0, argv=0x157c07a8, blockptr=0x0) at vm.c:881
#58 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#59 0x00007f9875bac62c in proc_call (argc=0, argv=0x157c07a8, procval=136275920) at proc.c:724
#60 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x133927d0, reg_cfp=0x158c0320, ci=<value optimized out>) at vm_insnhelper.c:1510
#61 0x00007f9875cd0d32 in vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:1025
#62 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#63 0x00007f9875cd7089 in vm_call0_body (th=0x133927d0, ci=0x7f9868f463f0, argv=<value optimized out>) at vm_eval.c:171
#64 0x00007f9875cd7597 in vm_call0 (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:50
#65 rb_vm_call (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:247
#66 0x00007f9875ba94d8 in rb_method_call_with_block (argc=0, argv=0x4022cd8, method=47064120, pass_procval=8) at proc.c:1831
#67 0x00007f9875baa93a in bmcall (args=67251400, method=47064120, argc=0, argv=<value optimized out>, passed_proc=8) at proc.c:2306
#68 0x00007f9875ccc3e5 in vm_yield_with_cfunc (th=0x133927d0, block=0xf8dac20, self=<value optimized out>, argc=0, argv=0x157c0688, blockargptr=<value optimized out>) at vm_insnhelper.c:2136
#69 0x00007f9875cd4846 in vm_invoke_block (th=0x133927d0, initial=<value optimized out>) at vm_insnhelper.c:2386
#70 vm_exec_core (th=0x133927d0, initial=<value optimized out>) at insns.def:1064
#71 0x00007f9875cd656f in vm_exec (th=0x133927d0) at vm.c:1427
#72 0x00007f9875cdb914 in invoke_block_from_c (th=0x133927d0, block=<value optimized out>, self=130918320, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=103117240) at vm.c:817
#73 0x00007f9875cdbc63 in vm_invoke_proc (th=0x133927d0, proc=0x138ee8a0, self=130918320, defined_class=103117240, argc=0, argv=0x2ce1930, blockptr=0x0) at vm.c:881
#74 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#75 0x00007f9875ced51e in thread_start_func_2 (th=0x133927d0, stack_start=0x7f9868f48000) at thread.c:535
#76 0x00007f9875cf3410 in thread_start_func_1 (th_ptr=0x133927d0) at thread_pthread.c:846
#77 0x00007f98759169d1 in start_thread () from /lib64/libpthread.so.0
#78 0x00007f9874d9c8fd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f9868e46700 (LWP 20567)):
#0 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
#1 0x00007f986b8ae35f in pqSocketCheck () from /opt/gitlab/embedded/lib/libpq.so.5
#2 0x00007f986b8ae3d0 in pqWaitTimed () from /opt/gitlab/embedded/lib/libpq.so.5
#3 0x00007f986b8adb49 in PQgetResult () from /opt/gitlab/embedded/lib/libpq.so.5
#4 0x00007f986b8addc8 in PQexecFinish () from /opt/gitlab/embedded/lib/libpq.so.5
#5 0x00007f986badd759 in gvl_PQexecPrepared_skeleton (data=<value optimized out>) at gvl_wrappers.c:9
#6 0x00007f9875cf14a2 in call_without_gvl (func=0x7f986badd730 <gvl_PQexecPrepared_skeleton>, data1=0x7f9868e41240, ubf=<value optimized out>, data2=<value optimized out>, fail_if_interrupted=0) at thread.c:1270
#7 0x00007f986badd2a9 in gvl_PQexecPrepared (conn=<value optimized out>, stmtName=<value optimized out>, nParams=<value optimized out>, paramValues=<value optimized out>, paramLengths=<value optimized out>, paramFormats=<value optimized out>, resultFormat=0) at gvl_wrappers.c:10
#8 0x00007f986bae471c in pgconn_exec_prepared (argc=<value optimized out>, argv=<value optimized out>, self=288878240) at pg_connection.c:1409
#9 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159bebc0, ci=<value optimized out>) at vm_insnhelper.c:1510
#10 0x00007f9875cd0d32 in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:1025
#11 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#12 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=133186400, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=108929120) at vm.c:817
#13 0x00007f9875cdbc63 in vm_invoke_proc (th=0x134e7960, proc=0x7f9800468940, self=133186400, defined_class=108929120, argc=0, argv=0x158c0f28, blockptr=0x0) at vm.c:881
#14 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#15 0x00007f9875bac62c in proc_call (argc=0, argv=0x158c0f28, procval=133230720) at proc.c:724
#16 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159bf0c0, ci=<value optimized out>) at vm_insnhelper.c:1510
#17 0x00007f9875cd0d32 in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:1025
#18 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#19 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=139879200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74521920) at vm.c:817
#20 0x00007f9875cdbc63 in vm_invoke_proc (th=0x134e7960, proc=0x7f98342d9fd0, self=139879200, defined_class=74521920, argc=1, argv=0x158c0ec0, blockptr=0x0) at vm.c:881
#21 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#22 0x00007f9875bac62c in proc_call (argc=1, argv=0x158c0ec0, procval=127615440) at proc.c:724
#23 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159bf1b0, ci=<value optimized out>) at vm_insnhelper.c:1510
#24 0x00007f9875cd131e in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:996
#25 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#26 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=136860440, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x7a5d2f8, defined_class=109825840) at vm.c:817
#27 0x00007f9875cdc1b6 in vm_yield_with_cref (under=<value optimized out>, self=136860440, values=128308040) at vm.c:848
#28 yield_under (under=<value optimized out>, self=136860440, values=128308040) at vm_eval.c:1529
#29 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159bf610, ci=<value optimized out>) at vm_insnhelper.c:1510
#30 0x00007f9875cdd373 in vm_call_cfunc (th=0x134e7960, cfp=0x159bf610, ci=0x46f4250) at vm_insnhelper.c:1600
#31 vm_call_method (th=0x134e7960, cfp=0x159bf610, ci=0x46f4250) at vm_insnhelper.c:1788
#32 0x00007f9875cd131e in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:996
#33 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#34 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=69209760, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522600) at vm.c:817
#35 0x00007f9875cdbc63 in vm_invoke_proc (th=0x134e7960, proc=0x7f9834242990, self=69209760, defined_class=74522600, argc=2, argv=0x158c0cc8, blockptr=0x0) at vm.c:881
#36 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#37 0x00007f9875bac62c in proc_call (argc=2, argv=0x158c0cc8, procval=127000200) at proc.c:724
#38 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159bf6b0, ci=<value optimized out>) at vm_insnhelper.c:1510
#39 0x00007f9875cd0d32 in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:1025
#40 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#41 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=74523200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74523160) at vm.c:817
#42 0x00007f9875cdbc63 in vm_invoke_proc (th=0x134e7960, proc=0x7f9834791ae0, self=74523200, defined_class=74523160, argc=1, argv=0x158c0c70, blockptr=0x0) at vm.c:881
#43 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#44 0x00007f9875bac62c in proc_call (argc=1, argv=0x158c0c70, procval=126999680) at proc.c:724
#45 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159bf750, ci=<value optimized out>) at vm_insnhelper.c:1510
#46 0x00007f9875cd131e in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:996
#47 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#48 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=126770280, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522080) at vm.c:817
#49 0x00007f9875cdc58d in vm_yield (val=126999680) at vm.c:856
#50 rb_yield_0 (val=126999680) at vm_eval.c:944
#51 rb_yield (val=126999680) at vm_eval.c:954
#52 0x00007f9875b592c7 in rb_ary_each (array=<value optimized out>) at array.c:1796
#53 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159bf7f0, ci=<value optimized out>) at vm_insnhelper.c:1510
#54 0x00007f9875cd131e in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:996
#55 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#56 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=176433920, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=57770080) at vm.c:817
#57 0x00007f9875cdbc63 in vm_invoke_proc (th=0x134e7960, proc=0x7f98000cc1c0, self=176433920, defined_class=57770080, argc=0, argv=0x158c07b8, blockptr=0x0) at vm.c:881
#58 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#59 0x00007f9875bac62c in proc_call (argc=0, argv=0x158c07b8, procval=136674360) at proc.c:724
#60 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x134e7960, reg_cfp=0x159c0330, ci=<value optimized out>) at vm_insnhelper.c:1510
#61 0x00007f9875cd0d32 in vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:1025
#62 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#63 0x00007f9875cd7089 in vm_call0_body (th=0x134e7960, ci=0x7f9868e453f0, argv=<value optimized out>) at vm_eval.c:171
#64 0x00007f9875cd7597 in vm_call0 (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:50
#65 rb_vm_call (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:247
#66 0x00007f9875ba94d8 in rb_method_call_with_block (argc=0, argv=0x44565c0, method=47079440, pass_procval=8) at proc.c:1831
#67 0x00007f9875baa93a in bmcall (args=71656880, method=47079440, argc=0, argv=<value optimized out>, passed_proc=8) at proc.c:2306
#68 0x00007f9875ccc3e5 in vm_yield_with_cfunc (th=0x134e7960, block=0x12eecb90, self=<value optimized out>, argc=0, argv=0x158c0698, blockargptr=<value optimized out>) at vm_insnhelper.c:2136
#69 0x00007f9875cd4846 in vm_invoke_block (th=0x134e7960, initial=<value optimized out>) at vm_insnhelper.c:2386
#70 vm_exec_core (th=0x134e7960, initial=<value optimized out>) at insns.def:1064
#71 0x00007f9875cd656f in vm_exec (th=0x134e7960) at vm.c:1427
#72 0x00007f9875cdb914 in invoke_block_from_c (th=0x134e7960, block=<value optimized out>, self=130917240, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=103117240) at vm.c:817
#73 0x00007f9875cdbc63 in vm_invoke_proc (th=0x134e7960, proc=0x13931330, self=130917240, defined_class=103117240, argc=0, argv=0x2ce52d8, blockptr=0x0) at vm.c:881
#74 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#75 0x00007f9875ced51e in thread_start_func_2 (th=0x134e7960, stack_start=0x7f9868e47000) at thread.c:535
#76 0x00007f9875cf3410 in thread_start_func_1 (th_ptr=0x134e7960) at thread_pthread.c:846
#77 0x00007f98759169d1 in start_thread () from /lib64/libpthread.so.0
#78 0x00007f9874d9c8fd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f9868a42700 (LWP 20571)):
#0 0x00007f9874d930d3 in poll () from /lib64/libc.so.6
#1 0x00007f986b8ae35f in pqSocketCheck () from /opt/gitlab/embedded/lib/libpq.so.5
#2 0x00007f986b8ae3d0 in pqWaitTimed () from /opt/gitlab/embedded/lib/libpq.so.5
#3 0x00007f986b8adb49 in PQgetResult () from /opt/gitlab/embedded/lib/libpq.so.5
#4 0x00007f986b8addc8 in PQexecFinish () from /opt/gitlab/embedded/lib/libpq.so.5
#5 0x00007f986badd759 in gvl_PQexecPrepared_skeleton (data=<value optimized out>) at gvl_wrappers.c:9
#6 0x00007f9875cf14a2 in call_without_gvl (func=0x7f986badd730 <gvl_PQexecPrepared_skeleton>, data1=0x7f9868a3d330, ubf=<value optimized out>, data2=<value optimized out>, fail_if_interrupted=0) at thread.c:1270
#7 0x00007f986badd2a9 in gvl_PQexecPrepared (conn=<value optimized out>, stmtName=<value optimized out>, nParams=<value optimized out>, paramValues=<value optimized out>, paramLengths=<value optimized out>, paramFormats=<value optimized out>, resultFormat=0) at gvl_wrappers.c:10
#8 0x00007f986bae471c in pgconn_exec_prepared (argc=<value optimized out>, argv=<value optimized out>, self=286183880) at pg_connection.c:1409
#9 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dbec00, ci=<value optimized out>) at vm_insnhelper.c:1510
#10 0x00007f9875cd0d32 in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:1025
#11 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#12 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=142021040, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=108929120) at vm.c:817
#13 0x00007f9875cdbc63 in vm_invoke_proc (th=0x1408a8e0, proc=0x7f97d891b190, self=142021040, defined_class=108929120, argc=0, argv=0x15cc0f68, blockptr=0x0) at vm.c:881
#14 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#15 0x00007f9875bac62c in proc_call (argc=0, argv=0x15cc0f68, procval=142017600) at proc.c:724
#16 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dbf100, ci=<value optimized out>) at vm_insnhelper.c:1510
#17 0x00007f9875cd0d32 in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:1025
#18 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#19 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=139879200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74521920) at vm.c:817
#20 0x00007f9875cdbc63 in vm_invoke_proc (th=0x1408a8e0, proc=0x7f98342d9fd0, self=139879200, defined_class=74521920, argc=1, argv=0x15cc0f00, blockptr=0x0) at vm.c:881
#21 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#22 0x00007f9875bac62c in proc_call (argc=1, argv=0x15cc0f00, procval=127615440) at proc.c:724
#23 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dbf1f0, ci=<value optimized out>) at vm_insnhelper.c:1510
#24 0x00007f9875cd131e in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:996
#25 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#26 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=136223800, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x7a0c510, defined_class=109825840) at vm.c:817
#27 0x00007f9875cdc1b6 in vm_yield_with_cref (under=<value optimized out>, self=136223800, values=127976920) at vm.c:848
#28 yield_under (under=<value optimized out>, self=136223800, values=127976920) at vm_eval.c:1529
#29 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dbf650, ci=<value optimized out>) at vm_insnhelper.c:1510
#30 0x00007f9875cd131e in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:996
#31 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#32 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=69209760, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522600) at vm.c:817
#33 0x00007f9875cdbc63 in vm_invoke_proc (th=0x1408a8e0, proc=0x7f9834242990, self=69209760, defined_class=74522600, argc=2, argv=0x15cc0d08, blockptr=0x0) at vm.c:881
#34 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#35 0x00007f9875bac62c in proc_call (argc=2, argv=0x15cc0d08, procval=127000200) at proc.c:724
#36 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dbf6f0, ci=<value optimized out>) at vm_insnhelper.c:1510
#37 0x00007f9875cd0d32 in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:1025
#38 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#39 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=74523200, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74523160) at vm.c:817
#40 0x00007f9875cdbc63 in vm_invoke_proc (th=0x1408a8e0, proc=0x7f9834791ae0, self=74523200, defined_class=74523160, argc=1, argv=0x15cc0cb0, blockptr=0x0) at vm.c:881
#41 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#42 0x00007f9875bac62c in proc_call (argc=1, argv=0x15cc0cb0, procval=126999680) at proc.c:724
#43 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dbf790, ci=<value optimized out>) at vm_insnhelper.c:1510
#44 0x00007f9875cd131e in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:996
#45 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#46 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=126770280, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=74522080) at vm.c:817
#47 0x00007f9875cdc58d in vm_yield (val=126999680) at vm.c:856
#48 rb_yield_0 (val=126999680) at vm_eval.c:944
#49 rb_yield (val=126999680) at vm_eval.c:954
#50 0x00007f9875b592c7 in rb_ary_each (array=<value optimized out>) at array.c:1796
#51 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dbf830, ci=<value optimized out>) at vm_insnhelper.c:1510
#52 0x00007f9875cd131e in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:996
#53 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#54 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=176433920, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=57770080) at vm.c:817
#55 0x00007f9875cdbc63 in vm_invoke_proc (th=0x1408a8e0, proc=0x7f97d8060710, self=176433920, defined_class=57770080, argc=0, argv=0x15cc07f8, blockptr=0x0) at vm.c:881
#56 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#57 0x00007f9875bac62c in proc_call (argc=0, argv=0x15cc07f8, procval=134933440) at proc.c:724
#58 0x00007f9875ccc0ea in vm_call_cfunc_with_frame (th=0x1408a8e0, reg_cfp=0x15dc0370, ci=<value optimized out>) at vm_insnhelper.c:1510
#59 0x00007f9875cd0d32 in vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:1025
#60 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#61 0x00007f9875cd7089 in vm_call0_body (th=0x1408a8e0, ci=0x7f9868a413f0, argv=<value optimized out>) at vm_eval.c:171
#62 0x00007f9875cd7597 in vm_call0 (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:50
#63 rb_vm_call (th=<value optimized out>, recv=<value optimized out>, id=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, me=<value optimized out>, defined_class=103117400) at vm_eval.c:247
#64 0x00007f9875ba94d8 in rb_method_call_with_block (argc=0, argv=0x340f400, method=47127800, pass_procval=8) at proc.c:1831
#65 0x00007f9875baa93a in bmcall (args=54588400, method=47127800, argc=0, argv=<value optimized out>, passed_proc=8) at proc.c:2306
#66 0x00007f9875ccc3e5 in vm_yield_with_cfunc (th=0x1408a8e0, block=0x11f56af0, self=<value optimized out>, argc=0, argv=0x15cc06d8, blockargptr=<value optimized out>) at vm_insnhelper.c:2136
#67 0x00007f9875cd4846 in vm_invoke_block (th=0x1408a8e0, initial=<value optimized out>) at vm_insnhelper.c:2386
#68 vm_exec_core (th=0x1408a8e0, initial=<value optimized out>) at insns.def:1064
#69 0x00007f9875cd656f in vm_exec (th=0x1408a8e0) at vm.c:1427
#70 0x00007f9875cdb914 in invoke_block_from_c (th=0x1408a8e0, block=<value optimized out>, self=130912920, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0, defined_class=103117240) at vm.c:817
#71 0x00007f9875cdbc63 in vm_invoke_proc (th=0x1408a8e0, proc=0xfac6e80, self=130912920, defined_class=103117240, argc=0, argv=0x2cf09a8, blockptr=0x0) at vm.c:881
#72 0x00007f9875cdbe0a in rb_vm_invoke_proc (th=<value optimized out>, proc=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, blockptr=<value optimized out>) at vm.c:900
#73 0x00007f9875ced51e in thread_start_func_2 (th=0x1408a8e0, stack_start=0x7f9868a43000) at thread.c:535
#74 0x00007f9875cf3410 in thread_start_func_1 (th_ptr=0x1408a8e0) at thread_pthread.c:846
#75 0x00007f98759169d1 in start_thread () from /lib64/libpthread.so.0
#76 0x00007f9874d9c8fd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f9876185700 (LWP 19973)):
#0 0x00007f987591a5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f9875ce9889 in native_cond_wait (cond=<value optimized out>, mutex=<value optimized out>) at thread_pthread.c:334
#2 0x00007f9875cede1b in native_sleep (th=0x238e6a0, timeout_tv=0x0) at thread_pthread.c:1102
#3 0x00007f9875cefa58 in rb_thread_terminate_all () at thread.c:447
#4 0x00007f9875ba528d in ruby_cleanup (ex=6) at eval.c:188
#5 0x00007f9875ba5575 in ruby_run_node (n=0x37615a8) at eval.c:310
#6 0x00000000004008db in main (argc=32, argv=0x7fffb0614d28) at main.c:36
(gdb)
```
### Tail from sidekiq log
```
2016-07-12_21:46:00.20028 2016-07-12T21:46:00.200Z 19973 TID-b7f7k INFO: Shutting down
2016-07-12_21:46:00.20049 2016-07-12T21:46:00.200Z 19973 TID-hb9qc INFO: Scheduler exiting...
2016-07-12_21:46:00.20073 2016-07-12T21:46:00.200Z 19973 TID-b7f7k INFO: Terminating quiet workers
2016-07-12_21:46:00.20098 2016-07-12T21:46:00.200Z 19973 TID-12ywx8 INFO: Scheduler exiting...
2016-07-12_21:46:00.70137 2016-07-12T21:46:00.701Z 19973 TID-b7f7k INFO: Pausing to allow workers to finish...
2016-07-12_21:46:03.70275 2016-07-12T21:46:03.702Z 19973 TID-b7f7k WARN: Terminating 6 busy worker threads
2016-07-12_21:46:03.70391 2016-07-12T21:46:03.703Z 19973 TID-b7f7k WARN: Work still in progress [#<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DeleteUserWorker\",\"args\":[1,40],\"retry\":true,\"queue\":\"default\",\"jid\":\"1266976cd7b666c721bc017d\",\"created_at\":1468256832.5222852,\"enqueued_at\":1468256832.5224586}">, #<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DeleteUserWorker\",\"args\":[69,40],\"retry\":true,\"queue\":\"default\",\"jid\":\"2b4f661c2b38a88c6f007be0\",\"created_at\":1468215212.9621508,\"enqueued_at\":1468215212.9624505}">, #<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DeleteUserWorker\",\"args\":[1,40],\"retry\":true,\"queue\":\"default\",\"jid\":\"19e2dc7a9942954670cf16e2\",\"created_at\":1468256844.0362723,\"enqueued_at\":1468256844.0364516}">, #<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DeleteUserWorker\",\"args\":[69,40],\"retry\":true,\"queue\":\"default\",\"jid\":\"f3699f7337fde1096dda2386\",\"created_at\":1468205606.417702,\"enqueued_at\":1468205606.4178174}">, #<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DeleteUserWorker\",\"args\":[69,40],\"retry\":true,\"queue\":\"default\",\"jid\":\"f46ca007b42d016228226ed7\",\"created_at\":1468354054.0193346,\"enqueued_at\":1468354054.0195577}">, #<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DeleteUserWorker\",\"args\":[69,40],\"retry\":true,\"queue\":\"default\",\"jid\":\"f5e5d8912660fc7f4bb6a5f4\",\"created_at\":1468205793.516869,\"enqueued_at\":1468205793.5169833}">]
2016-07-12_21:46:03.70473 2016-07-12T21:46:03.704Z 19973 TID-b7f7k INFO: Pushed 6 jobs back to Redis
2016-07-12_21:46:03.70573 2016-07-12T21:46:03.705Z 19973 TID-b7f7k INFO: Bye!
tail: inotify cannot be used, reverting to polling
```
### PG blocked/blocking queries
```
blocked_pid | blocked_user | blocking_pid | blocking_user | blocked_statement | current_statement_in_blocking_process | blocked_application | blocking_application
-------------+--------------+--------------+---------------+-------------------------------------------------------+-------------------------------------------------------+-------------------------------------------+-------------------------------------------
20645 | gitlab | 20584 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20650 | gitlab | 20584 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20683 | gitlab | 20584 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20659 | gitlab | 20640 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20650 | gitlab | 20645 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20683 | gitlab | 20645 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20584 | gitlab | 20645 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20645 | gitlab | 20650 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20683 | gitlab | 20650 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20584 | gitlab | 20650 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20645 | gitlab | 20659 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20650 | gitlab | 20659 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20683 | gitlab | 20659 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20584 | gitlab | 20659 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20645 | gitlab | 20683 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20650 | gitlab | 20683 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
20584 | gitlab | 20683 | gitlab | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | DELETE FROM "namespaces" WHERE "namespaces"."id" = $1 | sidekiq 4.1.2 gitlab-rails [0 of 25 busy] | sidekiq 4.1.2 gitlab-rails [0 of 25 busy]
(17 rows)
```
***
/cc @marin @pcarranza @dblessing @stanhuEdited by 🤖 GitLab Bot 🤖