Skip to content

Use Redis pub/sub to watch for Geo proxy changes

Catalin Irimie requested to merge cat-geo-pubsub-redis into master

What does this MR do and why?

Instead of polling the internal API, this uses Redis and watches for the proxy config to be changed.

TL;DR from the code comments:

// The polling/pubsub flow is the following:
//
// - always sleep at least `eoProxyApiPollingInterval`
// - wait for Redis notifications (if available) that the data changes
// - get fresh data from the API
//
// Flow:
//   sleep 10s  <----------------------------------------------------------.
//        |                                                                ^ loop
//   listen to Redis notifications  <-. loop until different or Redis      |
//        |                           | inaccessible / not configured      |
//        |---------------------------^                                    |
//   fetch from API                                                        |
//        |                                                                |
//        |>-------------------------->------------------------------------^

This avoids an initial request to the internal API and should keep watching forever if Redis is configured, and proxying is not enabled.

If proxying is enabled, the key will be different, and watching will stop, resulting in a request to the API (refreshing the key as well). Refreshing the key happens in a Sidekiq worker as well for consistency (since, if the key is empty or configuration changes, it's possible that no-one hits the API to refresh it)

How to set up and validate locally

I've personally set up a GET Geo env, patched a Workhorse binary with extra logging and uploaded it, i.e.:

diff --git a/workhorse/internal/upstream/upstream.go b/workhorse/internal/upstream/upstream.go
index ae5ff0852d7..f200b6ff3bb 100644
--- a/workhorse/internal/upstream/upstream.go
+++ b/workhorse/internal/upstream/upstream.go
@@ -243,6 +243,7 @@ func (u *upstream) waitForRedisOrSleep() {
 
 	for {
 		result, err := u.geoProxyWatcher(u.geoDataHash, geoProxyApiPollingInterval)
+		log.WithFields(log.Fields{"result": result, "err": err}).Info("Got data from redis")
 
 		if err != nil {
 			return
@@ -266,6 +267,7 @@ func (u *upstream) waitForRedisOrSleep() {
 // Calls /api/v4/geo/proxy and sets up routes
 func (u *upstream) updateGeoProxyData() {
 	geoProxyData, err := u.APIClient.GetGeoProxyData()
+	log.WithFields(log.Fields{"data": geoProxyData}).Info("Got data from API")
 	if err != nil {
 		log.WithError(err).WithFields(log.Fields{"geoProxyBackend": u.geoProxyBackend}).Error("Geo Proxy: Unable to determine config. Is the internal Rails API available? Falling back on cached value.")
 		return

Then tested the various cases, changing the unified URL, enabling/disabling the FF for separate URLs etc.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Catalin Irimie

Merge request reports