Skip to content

Provide emptyDir for temp files for webservice [non-prod]

John Skarbek requested to merge jts/extra-dir-non-prod into master

What does this MR do?

  • The web fleet slowly gathers files from the Dependency Proxy feature which is not cleaning up after itself sometimes.
  • Until this feature is fixed, we need to provide a place to safely store files temporarily, with a limit to prevent the nodes servicing the web Deployment don't run the node out of disk space
  • This configures an emptyDir configuration with a limit of 10GB
  • After 10GB of data is written to this location, Kubernetes will intervene and kill the Pod
  • A limit of 10 was chosen after analysis on the web nodes:
    • Normally we only see a couple of 2GB files that are left on disk
    • We have roughly 80GB of usable space on the nodes that will service this workload
    • Desired pod counts per node is not entirely settled on, but when using the same numbers as the API, if we have 4 Pods and all fill this emptyDir, this leaves 50% of the volume of the node
    • I'd like to ensure if we decide to run more Pods, we have space available
    • If we decide to provide less Pods per node, we can quickly tune this number appropriately
  • The path was determined after looking at the rails code responsible for the Dependency Proxy feature
    • In this case, we perform a File.open(Tempfile.new, "wb")
    • In doing so, we learn that inside of a Pod, a directory created under the path /tmp/gitlab/ to host the temporary data

Addresses: gitlab-com/gl-infra/delivery#1864 (closed)

Author Check-list

Please read the Contributing document and once you do, complete the following:

  • Assign to the correct reviewer per the contributing document
  • Apply the correct metadata per the contributing document

Reviewer Check-list

  • Reviewed the diff jobs to confirm changes are as expected
  • No changes shown in the diffs not associated with this MR - This may require a rebase or further investigation
Edited by John Skarbek

Merge request reports