GitLab makes many unnecessary HEAD requests to Elasticsearch

Problem

As we've noticed from looking at logs we see GitLab is making many HEAD requests to the ES cluster.

The problem is in gitlab-elasticsearch-indexer startup. This performs a healthcheck.

You can confirm this by running the elastic/client_test.go when the cluster is not running:

$ export ELASTIC_CONNECTION_INFO='{"url":["http://localhost:9999"], "index_name":"gitlab-test"}'

$ go test -v ./elastic/client_test.go

...
    TestElasticClientIndexAndRetrieval: client_test.go:237:
        	Error Trace:	client_test.go:237
        	            				client_test.go:245
        	            				client_test.go:252
        	Error:      	Received unexpected error:
        	            	no Elasticsearch node available
        	            	github.com/olivere/elastic.init
        	            		/Users/dylan/go/pkg/mod/github.com/olivere/elastic@v6.2.24+incompatible/client.go:88
        	            	runtime.doInit
        	            		/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:5414
        	            	runtime.doInit
        	            		/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:5409
        	            	runtime.doInit
        	            		/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:5409
        	            	runtime.doInit
        	            		/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:5409
        	            	runtime.main
        	            		/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:190
        	            	runtime.goexit
        	            		/usr/local/Cellar/go/1.14.1/libexec/src/runtime/asm_amd64.s:1373
        	            	health check timeout: Head "http://localhost:9999": dial tcp 127.0.0.1:9999: connect: connection refused
        	            	github.com/olivere/elastic.(*Client).startupHealthcheck
        	            		/Users/dylan/go/pkg/mod/github.com/olivere/elastic@v6.2.24+incompatible/client.go:1181
        	            	github.com/olivere/elastic.DialContext
        	            		/Users/dylan/go/pkg/mod/github.com/olivere/elastic@v6.2.24+incompatible/client.go:363
        	            	github.com/olivere/elastic.NewClient
        	            		/Users/dylan/go/pkg/mod/github.com/olivere/elastic@v6.2.24+incompatible/client.go:198
        	            	gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/elastic.NewClient
        	            		/Users/dylan/workspace/gitlab-elasticsearch-indexer/elastic/client.go:122
        	            	gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/elastic.FromEnv
        	            		/Users/dylan/workspace/gitlab-elasticsearch-indexer/elastic/client.go:55
        	            	command-line-arguments_test.setupTestClient

For every project update to the repo there are 2 requests:

1. For updated blobs
2. For new commits

Solution

This healthcheck is probably unnecessary and due to the shortlived running of our gitlab-elasticsearch-indexer we may wish to look if we can just disable it and happily allow the bulk request to fail if the cluster is down for whatever reason.

The HEAD requests may be degrading performance but they are also likely contributing to tonnes of extra logging.