Set GITLAB_TEMPFILE_IMMEDIATE_UNLINK env var to '1'

Production Change

Change Summary

We have seen issues with our Rails application where large incoming requests are written to disk in temporary files and not cleaned up. https://gitlab.com/gitlab-org/gitlab/-/issues/324817 contains some more details here. This is particularly common on the API fleet.

gitlab-org/gitlab!57239 (merged) makes a change to the Ruby libraries in question to immediately unlink those temporary files. This means that while they will still consume disk space while the request is being processed (or at worst, while the Puma process is running), the space can be automatically reclaimed by the OS as the file isn't persisted beyond the lifetime of the Puma process.

This change is behind an environment variable - GITLAB_TEMPFILE_IMMEDIATE_UNLINK - as we are monkey-patching Puma, and so we can't use a feature flag. (We intend to propose the change upstream if it works for us.)

Change Details

  1. Services Impacted - API, git, and web
  2. Change Technician - @msmiley / @smcgivern
  3. Change Criticality - C3
  4. Change Type - changescheduled
  5. Change Reviewer - @msmiley / @smcgivern
  6. Due Date - 2021-04-20 19:00 UTC
  7. Time tracking - 120 minutes roll-out, 120 minutes rollback
  8. Downtime Component - None expected

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 10 mins

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 90 mins

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 10 mins

This will involve some manual testing. The commit descriptions in gitlab-org/gitlab!57239 (merged) have some detail, but essentially we can just try sending large-ish request bodies (say, 10 MiB) and see what happens. We'll either need to monitor all API nodes or figure out how to get the right one; if we use curl then we can use its --limit-rate option to slow down the transfer.

For 'seeing what happens', we can compare when unlink or unlinkat system calls happen compared to the current state. We can also inspect /tmp directly and with a tool like inotifywatch.

One way to test is to run the following script on an API node:

test_tempfile_immediate_unlink.sh

This encapsulates the manual testing we did in the staging environment. It makes an API call while tracing temp file creation and deletion. Before the change, tempfiles will have a non-zero age. After the change, tempfiles from Puma and Rack will consistently have a zero-second age, due to the new immediate-unlink behavior.

  • In the GitLab web UI, create yourself a disposable short-lived Personal Access Token (PAT):
  • Run the test script as shown below from any of the API hosts: test_tempfile_immediate_unlink.sh It creates a test file to upload and makes an API call to the uploads endpoint of the given disposable project. While running that API call, it traces file creation and deletion events, showing the age and deletion timestamp for short-lived files. Confirm that after applying this change, the temp files created by Puma and Rack have the expected 0-second age.
$ export GITLAB_API_TOKEN=<REDACTED>
$ export GITLAB_API_DOMAIN=gitlab.com
$ export PROJECT_ID=21158863
$ ./test_tempfile_immediate_unlink.sh

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 90

Monitoring

Key metrics to observe

Summary of infrastructure changes

  • Does this change introduce new compute instances?
  • Does this change re-size any existing compute instances?
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • There are currently no active incidents.
Edited by Matt Smiley