Provide emptyDir for temp files for webservice [non-prod]
What does this MR do?
- The web fleet slowly gathers files from the Dependency Proxy feature which is not cleaning up after itself sometimes.
- Until this feature is fixed, we need to provide a place to safely store files temporarily, with a limit to prevent the nodes servicing the web Deployment don't run the node out of disk space
- This configures an emptyDir configuration with a limit of 10GB
- After 10GB of data is written to this location, Kubernetes will intervene and kill the Pod
- A limit of 10 was chosen after analysis on the web nodes:
- Normally we only see a couple of 2GB files that are left on disk
- We have roughly 80GB of usable space on the nodes that will service this workload
- Desired pod counts per node is not entirely settled on, but when using the same numbers as the API, if we have 4 Pods and all fill this emptyDir, this leaves 50% of the volume of the node
- I'd like to ensure if we decide to run more Pods, we have space available
- If we decide to provide less Pods per node, we can quickly tune this number appropriately
- The path was determined after looking at the rails code responsible
for the Dependency Proxy feature
- In this case, we perform a File.open(Tempfile.new, "wb")
- In doing so, we learn that inside of a Pod, a directory created under the path /tmp/gitlab/ to host the temporary data
Addresses: gitlab-com/gl-infra/delivery#1864 (closed)
Author Check-list
Please read the Contributing document and once you do, complete the following:
-
Assign to the correct reviewer per the contributing document -
Apply the correct metadata per the contributing document
Reviewer Check-list
-
Reviewed the diff jobs to confirm changes are as expected -
No changes shown in the diffs not associated with this MR - This may require a rebase or further investigation
Edited by John Skarbek