Geo: Faster event processing by caching inside GeoNode#projects_include?
From !32364 (comment 344006796):
The code looks good but I wonder if this is too expensive to calculate things like that when we consume each event.
@vsizov Good point. It took me surprisingly long to find how we skip events currently, for things that are excluded from selective sync.
https://gitlab.com/gitlab-org/gitlab/-/blob/v12.10.6-ee/ee/lib/gitlab/geo/log_cursor/daemon.rb#L155
It appears we already run this query for all events that have a
project_id
except project deletions:
- hashed storage attachments
- hashed storage migrated
- repo created/updated/renamed
- repo reset checksum
I think since we want events for Blobs, then it's ok to do the same. As always, we need to keep an eye on performance. Dogfooding production GitLab.com should help us here.
Hm, interesting. I'm realizing now that it's the only way to go as we create events for all cases on primary, we don't take into consideration secondaries' settings, which is OK but this also means that every secondary is responsible for the event selection.
LGTM
It's probably a good place for optimization. For example, we could cache project ids and invalidate the cache when selective sync setting are changed.
Many events need to check if a particular project is included/excluded by selective sync. I don't have hard data, but it's conceivable on a busy instance to have hundreds of events per second calling https://gitlab.com/gitlab-org/gitlab/-/blob/v12.10.6-ee/ee/app/models/geo_node.rb#L279.
Possible proposal
We could potentially cache this line inside of GeoNode#projects_include?
https://gitlab.com/gitlab-org/gitlab/-/blob/v12.10.6-ee/ee/app/models/geo_node.rb#L282 by project_id
.
We should be sure that caching (and implementing invalidation) would be beneficial. Related article: https://mailchi.mp/railsspeed/do-you-have-to-add-caching-to-scale-well?e=ec6b6a1eae
What is the performance of these queries? https://gitlab.com/gitlab-org/gitlab/-/blob/v12.10.6-ee/ee/app/models/geo_node.rb#L402-408 What would the cache hitrate be?