Potential memory leak in the Docker Registry
Summary
Supposedly this is a known (through internal conversation), but I've been unable to find an existing issue.
The Docker Registry shows signs of memory leakage.
Sometimes the process will panic:
2018-11-30_11:23:58.61295 fatal error: runtime: out of memory
2018-11-30_11:23:58.61998
2018-11-30_11:23:58.62072 runtime stack:
2018-11-30_11:23:58.62379 runtime.throw(0xd10b14, 0x16)
2018-11-30_11:23:58.62430 /usr/local/go/src/runtime/panic.go:616 +0x81
2018-11-30_11:23:58.62431 runtime.sysMap(0xc5a1500000, 0x800000, 0x0, 0x123b938)
2018-11-30_11:23:58.62432 /usr/local/go/src/runtime/mem_linux.go:216 +0x20a
2018-11-30_11:23:58.62433 runtime.(*mheap).sysAlloc(0x12207c0, 0x800000, 0x0)
2018-11-30_11:23:58.62433 /usr/local/go/src/runtime/malloc.go:470 +0xd4
2018-11-30_11:23:58.62433 runtime.(*mheap).grow(0x12207c0, 0x400, 0x0)
2018-11-30_11:23:58.62433 /usr/local/go/src/runtime/mheap.go:907 +0x60
2018-11-30_11:23:58.62434 runtime.(*mheap).allocSpanLocked(0x12207c0, 0x400, 0x123b948, 0xc451a3f4a0)
2018-11-30_11:23:58.62434 /usr/local/go/src/runtime/mheap.go:820 +0x301
2018-11-30_11:23:58.62434 runtime.(*mheap).alloc_m(0x12207c0, 0x400, 0x7ffed7990101, 0x7fe4957fae20)
2018-11-30_11:23:58.62434 /usr/local/go/src/runtime/mheap.go:686 +0x118
2018-11-30_11:23:58.62435 runtime.(*mheap).alloc.func1()
2018-11-30_11:23:58.62435 /usr/local/go/src/runtime/mheap.go:753 +0x4d
2018-11-30_11:23:58.62435 runtime.(*mheap).alloc(0x12207c0, 0x400, 0x7fe495010101, 0x43d5e5)
2018-11-30_11:23:58.62435 /usr/local/go/src/runtime/mheap.go:752 +0x8a
2018-11-30_11:23:58.62435 runtime.largeAlloc(0x800000, 0x101, 0xc400000001)
2018-11-30_11:23:58.62436 /usr/local/go/src/runtime/malloc.go:826 +0x94
2018-11-30_11:23:58.62436 runtime.mallocgc.func1()
2018-11-30_11:23:58.62436 /usr/local/go/src/runtime/malloc.go:721 +0x46
2018-11-30_11:23:58.62436 runtime.systemstack(0x7fe400000000)
2018-11-30_11:23:58.62437 /usr/local/go/src/runtime/asm_amd64.s:409 +0x79
2018-11-30_11:23:58.62437 runtime.mstart()
2018-11-30_11:23:58.62437 /usr/local/go/src/runtime/proc.go:1175
Steps to reproduce
Run a docker registry with a couple of TB of images for awhile with an average request rate of 150 per minute.
What is the current bug behavior?
The Docker Registry will crash
What is the expected correct behavior?
The Docker Registry should be able to clean itself up over time preventing a failure.
Relevant logs and/or screenshots
See above
I have the logs for 1 full day. These contain customer information and are only available on request.
Output of checks
This bug happens on GitLab.com