docs(v2/httpserver): add readiness check best practices

Elliot Forbesrequested to merge
docs/httpserver-readiness-best-practices into master

Summary

  • Adds readiness check best practices documentation to v2/httpserver/README.md, covering liveness vs readiness semantics, startup probes, the cascading failure anti-pattern, and Kubernetes probe configuration guidance
  • Implements a dedicated probe port (default :9090) that serves /-/liveness, /-/readiness, and /-/metrics — the main application port now serves only application traffic
  • Updates all Kubernetes probe YAML examples to reference port 9090

What changed

v2/httpserver/server.go

  • New Config.ProbeAddr field (default ":9090") — configures the dedicated probe server address; use ":0" in tests
  • New Server.ProbeAddr() method — returns the bound probe address after Start
  • NewWithConfig builds a private probeSrv *http.Server with its own http.ServeMux routing /-/liveness, /-/readiness, and /-/metrics (no middleware applied)
  • Start binds both listeners sequentially; if the probe bind fails after the app bind succeeds, the app listener is closed (no leaked resources)
  • Shutdown drains the app server first, then shuts down the probe server — /-/readiness stays observable during the drain window
  • ServeHTTP simplified: health paths removed, application traffic only; requests to /-/liveness or /-/readiness on the main port return 404

v2/httpserver/server_test.go

  • All existing health tests (TestLiveness_*, TestReadiness_*, TestAddReadinessCheck_IsChainable) rewritten to use a real probe listener on :0 via a startProbeServer helper
  • New table-driven tests: TestProbeAddr_EmptyBeforeStart, TestProbeServer_StartsAndReturnsAddr, TestProbeServer_Start_FailsIfProbeAddrInvalid, TestProbeServer_ServesHealthEndpoints, TestProbeServer_AppPortNoLongerServesProbes

v2/httpserver/README.md

  • Added ### Liveness, ### Startup, ### Readiness, ### Writing safe readiness checks subsections
  • Added startup probe guidance with failureThreshold: 24 (2 min grace) and explanation of when to use a custom startup endpoint
  • Added cascading failure anti-pattern section with numbered failure sequence
  • Added #### What to check, #### What not to check, #### Kubernetes probe configuration, #### Liveness probes subsections
  • Health endpoints table updated with Port column (:9090)
  • All Kubernetes probe YAML examples updated to port 9090
  • ProbeAddr added to the Configuration section example
  • Security note added: probe port can be restricted to internal cluster traffic independently of the app port
  • Prose reflowed to one sentence per line; startup probe wording clarified per review feedback

Test plan

  • CI pipeline passes (go test ./... in v2/)
  • /-/liveness and /-/readiness return expected responses on the probe port (:9090 default, :0 in tests)
  • /-/liveness and /-/readiness return 404 on the main app port
  • Server.ProbeAddr() returns empty string before Start, bound address after
  • Invalid ProbeAddr causes Start to return an error without leaking the app listener
  • Shutdown succeeds cleanly

Closes gitlab-org/quality/quality-engineering/team-tasks#4381 (closed)

Edited by Elliot Forbes

Merge request reports