Reduce worker runtime
Proposal
Some workers' runtime exceeds our target max runtime of 5 minutes, especially near the end of the month. We should discuss ways to parallelize jobs to help with scaling.
Regarding RemoveExpiredMembersWorker, @reprazent said:
This is not new to this merge request, but this won't scale infinitely. Looking at the durations of this worker the past 7 days it alrady sometimes exceeds our target maximum runtime of 5 minutes. Unsurprisingly, this happens for the jobs around the end of the month
😄 . | Nov 6, 2020 @ 00:11:18.643 | 77.577 | |----------------------------|--------| | Nov 5, 2020 @ 00:10:43.813 | 42.725 | | Nov 4, 2020 @ 00:10:33.113 | 31.982 | | Nov 3, 2020 @ 00:10:35.535 | 34.414 | | Nov 2, 2020 @ 00:10:35.844 | 34.666 | | Nov 1, 2020 @ 00:44:22.251 | 2,061.15 | | Oct 31, 2020 @ 00:22:27.774 | 746.681 |
https://log.gprd.gitlab.net/goto/31b13cb90c1d6a51ecf0c6726dec4810
As the feature gets used more, this job will keep taking longer until the amount of work to be done wont fit in a single job anymore. So we'll need to find a way to parallelize it.
Should we create an issue to discuss options?