Redis Hash cache for `Project#use_elasticsearch?`
What does this MR do?
The scope of the problem as defined in the original issue at #209854 (closed) is that we need to answer a boolean question "is ES enabled for this project?" for every update to any resource in that project. The source of truth to answer this question is in the Postgres database with a complex join query through parent namespaces. The details aren't important except to note that it's not cheap.
We are answering this question a lot for many different projects.
It's also worth noting that the answer changes very infrequently (eg. once a week/month) compared to how frequently it is answered (100 times per second).
The previous approach was reverted in !30012 (merged) due to performance problems with the caching being a very large list of ids being loaded from Redis too often which was slower than we'd hope and nearly as slow as the original DB querying it was replacing. More information in #214440 (closed) .
Further that other approach had an additional stale cache bug we hadn't solved where any newly created projects or subgroups in an indexed group where not being added to the list of indexed projects and thus would not end up in the index. This new MR does not suffer from that bug.
Then it introduces a new approach to caching of ApplicationSetting#elasticsearch_indexes_project? which does not suffer from the performance problems with the original approach. It is based on using a Redis hash to store the boolean value for every project.
This new MR does suffer from 1 stale cache bug where a project is moved to an indexed group but that was already present and we'll fix it in an new issue #214749 (closed) .
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team