Backend: Update AggregateLast30DayUsageService stop condition to reduce redundant processing
Summary
From #452545 (comment 1951203360):
There's more redundant processing happening than expected due to new catalog resources being added throughout the day. Since currrently a single job run can process all ~3k rows, it reprocesses them again from target_id=0
on the next run:
This isn't a significant concern at this time as the frequency of additions is relatively low. And counterintuitively, this redundancy will actually happen less frequently when we have many more catalog resources. This is because it will take more job runs to process all the rows, and on each following job run, it will have a fresh look at the database table but not restart back at target_id=0
.
I'll create a follow-up for this. I think the simplest change is to scope the done_processing?
query to only catalog resources that were added yesterday or earlier. To support this extended scope, the necessary index on catalog_resources
should be added/updated.
Proposal
Update the done_processing?
scope to only account for catalog resources added yesterday or older. In other words, we won't bother to run the service if we only have new catalog resources left to update.
UPDATE: It's easier to simply change the default value of catalog_resources.last_30_day_usage_count_updated_at
from 1970-01-01
to NOW()
. This means that done_processing?
will be unaffected by newly added resources because they'll already be seen as processed for today.