Skip to content

feat(datastore): add ability to cache repository objects in Redis

João Pereira requested to merge redis-cache-datastore into master

Related to #461 (closed).

This MR modifies the datastore package to include a new centralRepositoryCache type that will be our interface to the Redis-backed cache. For now, this will live alongside the existing singleRepositoryCache, which is being actively used, and this MR deprecates. To reduce risk, we're going to phase out the latter gradually, replacing it with the former.

We'll start by using this new Redis-backed cache for the "get repository details" GitLab v1 API operation only (that will be the next MR after this one). We'll then gradually expand its reach and finally, remove singleRepositoryCache and rename centralRepositoryCache to repositoryCache.

To avoid reinventing the wheel, I've relied on the following libs:

  • https://github.com/go-redis/redis: The Redis client lib. Already a dependency, no surprises here.

  • https://github.com/eko/gocache: A feature-rich Go cache library. There are a lot of great things here. Have a look at the description. The main selling points for me were:

    • Compatible with github.com/go-redis/redis;
    • Good code quality/coverage/activity;
    • Ability to chain caches. I'd like to have an L1 and L2 cache, where Redis is L2, and an in-memory cache will be L1 (not yet implemented). This will handle and abstract all of that from us;
    • Built-in Prometheus metrics collector;
    • Includes a marshaller so that we can transparently encode and decode struct to/from Redis. This is done using https://msgpack.org/ (through github.com/vmihailenco/msgpack).
  • https://github.com/alicebob/miniredis: Pure Go Redis server for tests. This is way easier and more lightweight than spinning up and configuring a Redis server for integration tests.

  • https://github.com/go-redis/redismock: Mocks for github.com/go-redis/redis. Together with the above, we should have a solid foundation for testing all Redis-backed features from now on. Have a look at the tests and test helpers introduced here to see what that looks like.

Edited by João Pereira

Merge request reports