Set SSL_CERT_DIR for embedded Go service - Elasticsearch Indexer
Summary
We tell our users to install custom SSL certificates into /opt/gitlab/embedded/ssl/certs/
. These certificates then get picked up by everything in omnibus-gitlab that uses OpenSSL.
However, we also have some Go programs (specifically Elasticsearch Indexer) in omnibus that use Go's own crypto/tls
library instead of OpenSSL (e.g. gitlab-workhorse, see gitlab-workhorse#177 (closed)). These programs will ignore /opt/gitlab/embedded/ssl/certs/
.
It turns out we can tell crypto/tls
about /opt/gitlab/embedded/ssl/certs/
by setting SSL_CERT_DIR=/opt/gitlab/embedded/ssl/certs/
. I suggest that we go through all our Runit services that spawn Go programs and add this setting to the default env
hash. It has been reported in gitlab-workhorse#177 (closed) that this works.
Steps to reproduce
When attempting to connect GitLab to Elasticsearch behind a certificate not signed by an authority Golang trusts, the connection fails.
What is the current bug behavior?
TLS Termination fails.
What is the expected correct behavior?
The connection to elasticsearch should succeed if the certificate chain is in the requested location.
Relevant logs
{
"severity": "WARN",
"time": "2020-01-28T17:20:48.413Z",
"error_class": "Gitlab::Elastic::Indexer::Error",
"error_message": "time=\"2020-01-28T17:19:50Z\" level=fatal msg=\"health check timeout: Head https://es_admin:***@hostnam:9200: x509: certificate signed by unknown authority: no Elasticsearch node available\"\n",
"context": "Job raised exception",
"jobstr": "{\"class\":\"ElasticCommitIndexerWorker\",\"args\":[5],\"retry\":2,\"queue\":\"elastic_commit_indexer\",\"jid\":\"1f5a925808afddc0af5342a9\",\"created_at\":1580231956.0210955,\"correlation_id\":\"7bf6e7a966c91c3285b3bbd126e05e60\",\"enqueued_at\":1580232043.2073886,\"error_message\":\"time=\\\"2020-01-28T17:19:50Z\\\" level=fatal msg=\\\"health check timeout: Head https://es_admin:***@hostnam:9200: x509: certificate signed by unknown authority: no Elasticsearch node available\\\"\\n\",\"error_class\":\"Gitlab::Elastic::Indexer::Error\",\"failed_at\":1580231961.4079654,\"retry_count\":1,\"retried_at\":1580231990.828342}",
"class": "ElasticCommitIndexerWorker",
"args": [
5
],
"retry": 2,
"queue": "elastic_commit_indexer",
"jid": "1f5a925808afddc0af5342a9",
"created_at": "2020-01-28T17:19:16.021Z",
"correlation_id": "7bf6e7a966c91c3285b3bbd126e05e60",
"enqueued_at": "2020-01-28T17:20:43.207Z",
"failed_at": "2020-01-28T17:19:21.407Z",
"retry_count": 1,
"retried_at": "2020-01-28T17:19:50.828Z",
"error_backtrace": [
"ee/lib/gitlab/elastic/indexer.rb:83:in `run_indexer!'",
"ee/lib/gitlab/elastic/indexer.rb:40:in `block in run'",
"ee/lib/gitlab/elastic/indexer.rb:39:in `each'",
"ee/lib/gitlab/elastic/indexer.rb:39:in `run'",
"ee/app/workers/elastic_commit_indexer_worker.rb:16:in `perform'",
"lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"
]
}