Potential Denial of Service via GET /api/v4/projects/<id>/languages
Hi,
I was attempting to gather language information across all our projects and discovered that doing so in a parallel manner slower than doing so serially. Upon investigation I discovered that GET /api/v4/projects/<id>/languages
caused commands to be run on the remote server to lookup and return the requested information. After further testing, here's why I think this should be considered a potential avenue for a denial of service attack:
- Small numbers of concurrent requests (6 in my test) take up a large amount of the available CPUs (75% on my 8 core system). It can also use up to a gigabyte of ram in the tests I ran.
- It appears that running more requests than there are CPUs causes an increase in response times to API calls, the web interface, and other parts of GitLab. In my case this caused the UI and other API requests to start taking 30-60 seconds to process with some requests timing out.
- Running a high number of requests continuously also started to cause
git push
over ssh to block until the load subsided. - Termination of these requests on the client side did not cause the remote commands to immediately terminate.
The code I wrote is attached for reference. The system we're running GitLab 11.5.1-ee on has more than enough memory and high performance disks attached so it is able to handle these requests fairly quickly under normal circumstances. After a certain point however it caused significant degradation in terms of performance, response handling, etc even in our deployment (8 core system, 30GB of ram, SSD backed, DB on separate host). For lower end systems I suspect this could result in I/O problems and potentially cause the system to begin terminating processes to free up memory in the worst case.
There are likely other factors that impact the effectiveness of this type of attack that I can't test or estimate the impact of on my own (how the server is configured, what projects require auth, project size, type of deployment [HA, Kubernetes, single server, autoscaling, etc]) which is another reason I'm reaching out as this could impact others more than an impacts us. Thanks and please let me know if you have any questions.
---Oliver Palmer