Upload diagnostic reports to GCS
Problem to solve
In #362900 (closed) we are looking to collect diagnostic reports in order to more easily debug production issues without help from SREs.
Once these reports are available on pod-local storage, we should publish them to a GCS bucket so that any team with read access to that bucket can download them for further analysis.
Proposal
For Puma, we cannot rely on uploads via Sidekiq, since they do not share storage in k8s. We will therefore have to initiate uploads directly from a Puma node.
Requirements
- Uploading potentially large files can be costly; any solution should have minimal impact on the ability of this node to serve user traffic.
- The uploader will scan a local directory for pending reports, and upload them to a GCS bucket.
- After a successful upload, local reports should be removed to free up storage.
Edited by Matthias Käppler