Cache Unleash clients payload

What does this MR do and why?

This MR introduces a caching layer to the Unleash API endpoint. The endpoint is used by GitLab Feature Flags feature.

This endpoint is currently nominated as the worst endpoint in terms of error budget. This is because of the two reasons 1) It takes time to read the feature flag data from PostgreSQL 2) It takes time to serialize the feature flag data for the unleash clients. You can find more in-depth analytics/metrics in this comment.

The key changes of MR are:

• Use present_cached helper to cache the serialized payload in Redis. This helper is also being used for the other intensive API endpoints.
• We add last_feature_flags_updated_at column to the feature_flags_clients table. This column is to track the latest modification time of feature flag payload. If a feature flag is created, updated or deleted, a new cache is generated again. An old cache will be expired within a day as the default expiry period.
• We don't proactively delete a cache since relying on Reids expire is considered as more reliable approach. Given that the change request rate to feature flags is low (You can read this issue for more information), we don't expect old caches will be pilled up.
• This change is behind cache_unleash_client_api feature flag, which is disabled by default.

Related #276406 (closed)

A few notes:

• Feature flags in the development of GitLab is unrelated to this problem/fix.

Screenshots or screen recordings

Requesting from unleash client

shinya@shinya-B550-VISION-D:~/workspace/thin-gdk/services/rails/src$ curl "http://local.gitlab.test:8181/api/v4/feature_flags/unleash/20/client/features?instance_id=Ai4rnFjPh4aNeiXJqb57&app_name=production" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   233  100   233    0     0    913      0 --:--:-- --:--:-- --:--:--   913
{
  "version": 1,
  "features": [
    {
      "name": "a_test",
      "description": "fasdfd",
      "enabled": true,
      "strategies": [
        {
          "name": "default",
          "parameters": {}
        }
      ]
    },
    {
      "name": "b_test",
      "description": "d",
      "enabled": true,
      "strategies": [
        {
          "name": "default",
          "parameters": {}
        }
      ]
    }
  ]
}

Confirm cache is created in Redis

local.gitlab.test:6379> GET cache:gitlab:Class:operations/feature_flags_clients/1:api_version:1:app_name:production:updated_at:1656394041
"\x04\bo: ActiveSupport::Cache::Entry\t:\x0b@valueI\"\x01\xe9{\"version\":1,\"features\":[{\"name\":\"a_test\",\"description\":\"fasdfd\",\"enabled\":true,\"strategies\":[{\"name\":\"default\",\"parameters\":{}}]},{\"name\":\"b_test\",\"description\":\"d\",\"enabled\":true,\"strategies\":[{\"name\":\"default\",\"parameters\":{}}]}]}\x06:\x06ET:\r@version0:\x10@created_atf\x171656395291.9646351:\x10@expires_inf\x0b8.64e4"

Requesting again after adding a flag

shinya@shinya-B550-VISION-D:~/workspace/thin-gdk/services/rails/src$ curl "http://local.gitlab.test:8181/api/v4/feature_flags/unleash/20/client/features?instance_id=Ai4rnFjPh4aNeiXJqb57&app_name=production" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   337  100   337    0     0   4109      0 --:--:-- --:--:-- --:--:--  4109
{
  "version": 1,
  "features": [
    {
      "name": "a_test",
      "description": "fasdfd",
      "enabled": true,
      "strategies": [
        {
          "name": "default",
          "parameters": {}
        }
      ]
    },
    {
      "name": "b_test",
      "description": "d",
      "enabled": true,
      "strategies": [
        {
          "name": "default",
          "parameters": {}
        }
      ]
    },
    {
      "name": "c-testflag",
      "description": "",
      "enabled": true,
      "strategies": [
        {
          "name": "default",
          "parameters": {}
        }
      ]
    }
  ]
}

Confirm a new cache is created in Redis

local.gitlab.test:6379> GET cache:gitlab:Class:operations/feature_flags_clients/1:api_version:1:app_name:production:updated_at:1656395430
"\x04\bo: ActiveSupport::Cache::Entry\t:\x0b@valueI\"\x02Q\x01{\"version\":1,\"features\":[{\"name\":\"a_test\",\"description\":\"fasdfd\",\"enabled\":true,\"strategies\":[{\"name\":\"default\",\"parameters\":{}}]},{\"name\":\"b_test\",\"description\":\"d\",\"enabled\":true,\"strategies\":[{\"name\":\"default\",\"parameters\":{}}]},{\"name\":\"c-testflag\",\"description\":\"\",\"enabled\":true,\"strategies\":[{\"name\":\"default\",\"parameters\":{}}]}]}\x06:\x06ET:\r@version0:\x10@created_atf\x171656395443.2668698:\x10@expires_inf\x0b8.64e4"

Confirm a cache is hit when an unleash client requests

^Cshinya@shinya-B550-VISION-D:~/workspace/thin-gdk$ devkitkat -i cli redis
INFO: This script is a custom script.
1656396046.157280 [0 127.0.0.1:49306] "get" "cache:gitlab:Class:operations/feature_flags_clients/1:api_version:1:app_name:production:updated_at:1656395430"
1656396056.562471 [0 127.0.0.1:49300] "get" "cache:gitlab:Class:operations/feature_flags_clients/1:api_version:1:app_name:production:updated_at:1656395430"
1656396057.820405 [0 127.0.0.1:49306] "get" "cache:gitlab:Class:operations/feature_flags_clients/1:api_version:1:app_name:production:updated_at:1656395430"

How to set up and validate locally

• Enable cache_unleash_client_api feature flag.
• Follow this doc to CRUD flags.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.