Support connecting to Redis Sentinel

This is a WIP from: gitlab-org/gitlab-ce#3355

We have many places loading redis configuration for different components (lot's of code duplication). This is also a good opportunity to remove references to "resque" that are still present on our codebase.

The idea here is to have a single place where configuration is loaded and all components who need it will get from there.

TODO:

• Create Gitlab::RedisConfig and concentrate all config loading logic there
• Remove duplicated code that loads resque.yml
• Handle Sentinel settings from our resque.yml file and fallback when needed
• Merge improvements from Gitlab::RedisConfig to Gitlab::Redis
• Refactor to the new Gitlab::Redis class
• Refactor mail_room.yml moving logic to separated class
• Make sure every component can connect with sentinel when defined
• Updated install documentation
• Open a MR to Omnibus to add dependent changes there (omnibus-gitlab!882 (merged))

---

How to test it with GDK

Sentinel is a separate daemon that monitors any amount of redis master servers and tries to achieve consensus when a failure is detect. Slave servers are automatically discovered by sentinel.

To setup the environment follow these steps

1. Update resque.yml with the example file resque.yml.example, uncomment and adapt the sentinel part

2. Duplicate redis folder to something like redis-slave

3. Configure different ports for both redis (you need to listen over TCP instead of UNIX socket)

4. On the slave redis.conf file you must add a SLAVEOF line pointing to the master like: slaveof 127.0.0.1 6379

5. Configure different ports in sentinel.conf and change mymaster to localhost and point in both to the configured redis port in redis folder, like the example below:

   port 26380
   sentinel monitor localhost 127.0.0.1 6379 1
   sentinel down-after-milliseconds localhost 10000
   sentinel config-epoch localhost 0
   sentinel leader-epoch localhost 0

6. You will need to run the 2 redis process and the 2 sentinel process, so change the Procfile like the example below:

   redis: redis-server /path/to/gitlab-development-kit/redis/redis.conf
   redis-slave: redis-server /path/to/gitlab-development-kit/redis-slave/redis.conf
   redis_sentinel: redis-server /path/to/gitlab-development-kit/redis/sentinel.conf --sentinel
   redis_sentinel_slave: redis-server /path/to/gitlab-development-kit/redis-slave/sentinel.conf --sentinel

To make sure your configuration is correct run rail console and try:

redis = Redis.new(Gitlab::Redis.params)
redis.info

# keep this screen open and try to simulate a failover below and run again after 10-15 seconds:
redis.info

you should get in the info a different port after a few seconds delay (the failover/reconnect time).

To simulate a failover on master redis, go to the terminal and run:

# port must match your master redis port
 redis-cli -h localhost -p 6379 DEBUG sleep 60

---

Troubleshooting

If you get an error like this one: Redis::CannotConnectError: No sentinels available. There may be something wrong with your configuration files or can be related to this: https://github.com/redis/redis-rb/issues/531 (PR that should make things better: https://github.com/redis/redis-rb/pull/534)

It's a bit rigid the way you have to config resque.yml and sentinel.conf otherwhise redis-rb will not work properly:

The hostname ('my-primary-redis') of the primary redis (in sentinel.conf) must match the one configure in resque.yml and it must be valid ex:

# sentinel.conf:
sentinel monitor my-primary-redis 127.0.0.1 6379 1
sentinel down-after-milliseconds my-primary-redis 10000
sentinel config-epoch my-primary-redis 0
sentinel leader-epoch my-primary-redis 0

# resque.yaml
development:
  url: redis://my-primary-redis:6378
  sentinels:
    -
      host: localhost
      port: 26380 # point to sentinel, not to redis port
    -
      host: localhost
      port: 26381 # point to sentinel, not to redis port

When in doubt, please read Redis Sentinel documentation