Use redis keyspace notifications in cleanup

Before raising this MR, consider whether the following are required, and complete if so:

• Unit tests
• Metrics
• Documentation update(s)

Description

This request aims to allow the cleanup of an S3 storage when using a Redis Index, through the use of keyspace notifications. Through the subscription to deletion and expiry event notifications, when they are enabled using the notify-keyspace-events of redis.conf (or using redsi-cli: redis-cli config set notify-keyspace-events KEA), digests are obtained from the 'key' contained in published events. These digests will be cleaned if their total reaches a specified amount (max_subscribed_messages), which is passed as a configuration option to the cleaner. Currently defaulting to 500. They will also be cleaned after a specified sleep_interval has passed since last deletions.

Changes proposed in this merge request:

• For a cleanup configuration that uses a RedisIndex, create a cleanup worker that subscribes to deletion and expiry Redis keyspace notifications, performing a bulk deletion of blobs from S3 storage when the above mentioned conditions have occurred.
• Changes to cleanup cmd and existing unit tests for the additional max_subscribed_messages parameter

Validation

• Construct a buildgrid docker-compose file with minio and redis services
• In the associated buildgrid Dockerfile, make use of these services when specifying the RedisIndex and associated storage

connections:
      - !sql-connection &sql
        connection-string: sqlite:///./example.db
        automigrate: yes
        connection-timeout: 15
      - !redis-connection &redisconn
        host: redis
        port: 6379

    storages:
      - !s3-storage &s3-storage
        bucket: buildgrid-cas-{digest[0]}{digest[1]}
        endpoint: http://minio:9000
        access-key: minioadmin
        secret-key: minioadmin
      - !redis-index &index
        storage: *s3-storage
        redis: *redisconn

• When the docker-compose is built and up, populate the S3 storage and RedisIndex with some blobs and keys. (The script I used is in the following snippet: $3644739)
• Check the blobs appear in the minio server
• Using redis-cli, check that keys are in redis
• Using a cleanup configuration, run the bgd cleanup startcommand. Example config:

server:
  - !channel
    port: 60052
    insecure-mode: true

description: |
  Config used for cleaning up

monitoring:
  enabled: false
  
instances:
  - name: test-clean
    description: |
      BGD cleanup

    connections:
        - !redis-connection &redis
          host: localhost
          port: 6379

    storages:
        - !s3-storage &s3-storage
          bucket: buildgrid-cas-{digest[0]}{digest[1]}
          endpoint: http://127.0.0.1:9000
          access-key: minioadmin
          secret-key: minioadmin
          versioned-deletes: true

        - !redis-index &index
          storage: *s3-storage
          redis: *redis

• Use redis-cli to configure keyspace notifications and then expire/delete some keys

127.0.0.1:6379> config set notify-keyspace-events KEA
OK
127.0.0.1:6379> del A:00df297f-32f9-4584-9ea1-6c7a99ddedec_3
(integer) 1
127.0.0.1:6379> expire A:0a1bea3d-5de5-4d73-b568-4d19227dd200_3 2
(integer) 1

• Check minio server that blobs have been deleted

Issues addressed

Closes / Resolves / Addresses (delete as appropriate) issue <e.g. repo-name#x>