Consider using persistent storage for flux source controller
Summary
During node rolling upgrades, source controller pod is drained, and it has to pull all the artifacts again.
This extra delay could be avoided by using persistent storage as describes in flux documentation:
related references
This will be probably needed to introduce per-unit timeout: !2383 (merged)
Otherwise various units may experience a large delay to reconcile when source-controller pod is re-scheduled.
We've observed such delays int that run for example: https://gitlab.com/sylva-projects/sylva-core/-/jobs/8831949769#L2798
/cc @tmmorin
Edited by Thomas Morin