Gitlab incredibly slow

My GitLab running on CentOS 7 is really slow, in fact, JavaScript files and sometimes some CSS time out. This happens on all pages, even on simple pages like the login screen. It can be accessed on http://git.favega.com/. It is running on one E5 core, with 1GB of RAM and 1.5GB of swap. Before you tell me, yes, I know swapping can make things slow, but gitlab is not being swapped. Every time I load a page, the CPU usage spikes to 100%, until nginx times unicorn out.