Skip to content

Nginx responds to health checks with correct content types

Adrien Kohlbecker requested to merge ak/proper-health-check-responses into master

While investigating gitlab#22975 (closed) I discovered that nginx is configured to respond to errors with static html pages.

In case of the health check endpoints, we actually want plain text (for /-/health) and json (for /-/readiness and /-/liveness).

The error codes currently handled in this manner are 404, 500 and 502 https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files%2Fgitlab-cookbooks%2Fgitlab%2Flibraries%2Fnginx.rb#L89

This merge request adds new location blocks overriding the error responses for the health check endpoints.

As a consequence, when gitlab-workhorse is down, nginx now correctly respond with json/text responses instead of HTML

➜ curl https://gitlab.example.com/-/readiness\?token\=TOKEN   
{"error": {"status_code": 502, "status": "Bad Gateway"}}

➜ curl https://gitlab.example.com/-/health\?token\=TOKEN
502 Bad Gateway

Note:

  • error_page in a location will reset any custom error configuration present in parent blocks
  • the location being .txt or .json will result in the correct content-type being set
  • Gitlab has custom html errors for 422 and 503 here https://gitlab.com/gitlab-org/gitlab-ee/tree/master/public but they are not used within nginx (only workhorse)
  • Since proxy_intercept_errors defaults to off, these blocks only apply to errors generated by nginx itself (eg. upstream is down)
  • Would love to add integration tests on this, is that possible?
  • Any tips on how to make these blocks more readable welcome
Edited by 🤖 GitLab Bot 🤖

Merge request reports