Skip to content

Update memcached jobspec to include Consul Connect proxy.

Created by: smarnach

This PR contains the changes I needed to make the Memcache deployment work with Consul Connect.

Currently Consul Connect isn't well integrated with Nomad, so it's necessary to run the proxy manually. This proved a bit tricky, since we want the memcached process to only listen on the loopback device, while the Consul Connect proxy of course needs to listen on the public network interface. I did not find any way to convince Nomad to use different IP addresses for different tasks, and this issue suggests that this is currently outright impossible. This made it impossible for me to register a correct Consul health check for the memcached process, which we need to make sure the process is operational and gets rescheduled if it is down. For now I completely removed the Consul service registration.

The documentation on using Consul Connect with Nomad states the following:

We recommend registering the source service as well since Consul can then know the health of the target service, which is used in determining if the proxy should receive requests.

However, the example they give does not do this, and after spending a lot of time trying I don't believe anymore this is even possible.

Another complication is that I needed to introduce different service names for each of the different memcached processes, since that's the only way to address them individually on the app server. The memcached deployment is sharded, so we need to create Consul Connect tunnels to each of the processes, but with a single service name it was only possible to create a tunnel to any one of the processes.

Other than that, the PR makes a small adjustment to have memcached log to stdout, so Nomad can pick up the logs, and they can be seen via the CLI or the UI.

Merge request reports