fix: fail at startup when TLS for the debug server cannot be set up

What does this MR do?

This is the follow-up asked for in the review of !2958 (merged):

the debug server falls back to plain HTTP on any TLS setup error. I think it should fail at startup like the main listener.

When http.debug.tls.enabled is true and TLS for the debug server cannot be set up, the registry now fails to start with the reason, as it does when http.tls cannot be set up, instead of serving the debug endpoints over plain HTTP. It is two commits:

1. configureMonitoring returns the error from getTLSConfig and closes the debug listener, where it used to log failed to configure TLS for debug server and keep serving the plain listener. When no certificate is set in http.debug.tls or http.tls, the error says that instead of no TLS config found.
2. The serve command calls configureMonitoring before the main listener starts and returns its error when it is one of these TLS errors. It used to call it in a goroutine that only logged the error (failed to configure monitoring service, skipping), so returning the error alone would have left the registry running without a debug server. Any other error, which today is only a debug address that cannot be bound, is still logged and skipped as before. Only monitoring.Start stays in the goroutine.

The tls subsection of debug in docs/configuration.md gets a short paragraph saying so.

Closes #2984 (closed)

Related to !2958 (merged)

What changes for a running registry

Only a registry with http.debug.addr set and http.debug.tls.enabled: true is affected.

• A registry whose debug TLS cannot be set up now exits at startup with configuring monitoring service: configuring TLS for the debug server: <reason>. The reasons are the ones http.tls already fails on: an unknown minimumtls, or a certificate, key or client CA file that cannot be read or parsed. For the debug server there is one more: enabled: true with no certificate in http.debug.tls or http.tls, for example when http.tls uses Let's Encrypt, which the debug server does not support.
• The Omnibus template writes no http.debug.tls, and the Helm chart refuses registry.debug.tls.enabled without a certificate for it when it renders, so for those deployments only a debug TLS setting the registry cannot use (a certificate or CA file it cannot load, or an unknown minimumTLS) changes anything. A chart deployment in that state does not become ready today either, because its liveness and readiness probes use HTTPS against a port that then serves plain HTTP.
• A debug address that cannot be bound is still logged as failed to configure monitoring service, skipping, and the registry runs without its debug server, as before. Only the TLS errors above stop startup. I can make that case fatal too, like a main address that cannot be bound, if you prefer.
• Once !2958 (merged) is merged, an unknown name in http.debug.tls.ciphersuites is one more of these errors, since it goes through getTLSConfig too. This branch is based on master, merges with !2958 (merged) without conflicts, and the registry and configuration tests pass on the merged tree.

A registry with one of these configurations that starts today will not start after this change. I have not added the breaking change label, since this is the behavior asked for in !2958 (merged), but I'm happy to if you think it needs it.

The tests, and how they fail on master

• TestConfigureMonitoring_DebugTLSSetupError: four setup errors (no certificate, certificate file missing, unknown minimumtls, client CA file missing). configureMonitoring must return the error, marked as one of the TLS errors that stop startup, and the debug address must be free again afterwards.
• TestConfigureMonitoring_DebugAddrInUseIsNotATLSError: a debug address already in use is still an error, but not one of the TLS errors that stop startup.
• TestServeCmd_DebugTLSSetupErrorStopsStartup: runs ServeCmd with http.debug.tls.enabled and no certificate. It must return the error with neither address left bound, and it fails as soon as the debug address answers plain HTTP.

On master at 15db9e60, Go 1.27.2, with a test-only declaration of errDebugTLS so that the tests compile:

--- FAIL: TestConfigureMonitoring_DebugTLSSetupError/certificate_file_missing
        Error:      Expected error with "configuring TLS for the debug server" in chain but got nil.
--- FAIL: TestConfigureMonitoring_DebugTLSSetupError/unknown_minimum_TLS_version
        Error:      Expected error with "configuring TLS for the debug server" in chain but got nil.
--- FAIL: TestConfigureMonitoring_DebugTLSSetupError/client_CA_file_missing
        Error:      Expected error with "configuring TLS for the debug server" in chain but got nil.
--- FAIL: TestConfigureMonitoring_DebugTLSSetupError/no_certificate
        Error:      Expected error with "configuring TLS for the debug server" in chain but got nil.
--- FAIL: TestServeCmd_DebugTLSSetupErrorStopsStartup
        Error:      the debug server fell back to plain HTTP
        Messages:   GET http://[::]:35357/debug/health answered "200 OK" with http.debug.tls.enabled set

TestConfigureMonitoring_DebugAddrInUseIsNotATLSError passes on master, since that case does not change.

With the first commit alone, the serve test fails because the registry keeps running without a debug server:

--- FAIL: TestServeCmd_DebugTLSSetupErrorStopsStartup (10.00s)
        Error:      the registry kept running although the debug server could not set up TLS

I also built cmd/registry on master and on this branch and started it with inmemory storage, http.addr: 127.0.0.1:5000, http.debug.addr: 127.0.0.1:5001 and http.debug.tls.enabled: true, with no certificate anywhere:

• master: still running after 4 seconds, curl http://127.0.0.1:5001/debug/health answers 200, and the log has "msg":"failed to configure TLS for debug server" with "error":"no TLS config found".
• This branch: exits with status 1 and configuring monitoring service: configuring TLS for the debug server: http.debug.tls.enabled is true, but no certificate is set in http.debug.tls or http.tls.

The same binaries with port 5001 already taken by another process both keep running and log failed to configure monitoring service, skipping with bind: address already in use, so that case is unchanged.

Checks I ran locally

• go test -tags continuous_profiler_stackdriver ./registry/ ./configuration/ -count=1 on Go 1.25.14, 1.26.8 and 1.27.2 (the CI matrix): pass. Each commit on its own, on Go 1.27.2: pass.
• The new tests with -count=10, and under -race with -count=3: pass. The whole registry package under -race reports a data race in TestGracefulShutdown on its cleanServerShutdown flag, on master as well; this MR does not touch that test.
• go vet -tags continuous_profiler_stackdriver ./registry/: clean.
• golangci-lint run ./registry/ ./configuration/ with golangci/golangci-lint:v2.13.1 and the project's .golangci.yml, on each commit: 0 issues. The config's build tags include cli_test, which leaves registry_test.go out, so I also ran it with --build-tags continuous_profiler_stackdriver,integration: 0 issues. golangci-lint fmt --diff: no changes.
• vale, markdownlint-cli2 and lychee --offline --include-fragments on docs/configuration.md with the lint-markdown image the docslint job uses: no findings.
• commitlint with @commitlint/config-conventional in node:lts-alpine on both commits: 0 problems.

Author checklist

• CODEOWNERS Review: This MR requires approval from at least one CODEOWNER per category/file.
  • If codeowners are absent or the change is urgent, any registry maintainer can temporarily disable CODEOWNERS reviews in the project settings. When doing so:
    • Add a comment on the MR justifying the bypass
    • Mention/CC the designated codeowners in the MR for async review
    • Re-enable CODEOWNERS reviews in project settings once the MR is merged
  • If you lack permissions to disable CODEOWNERS reviews, reach out to the Engineering Manager (@jaime) or Senior Engineering Manager (@crystalpoole) for assistance.
• Assign one of conventional-commit prefixes to the MR.
  • fix: Indicates a bug fix, triggers a patch release.
  • feat: Signals the introduction of a new feature, triggers a minor release.
  • perf: Focuses on performance improvements that don't introduce new features or fix bugs, triggers a patch release.
  • docs: Updates or changes to documentation. Does not trigger a release.
  • style: Changes that do not affect the code's functionality. Does not trigger a release.
  • refactor: Modifications to the code that do not fix bugs or add features but improve code structure or readability. Does not trigger a release.
  • test: Changes related to adding or modifying tests. Does not trigger a release.
  • chore: Routine tasks that don't affect the application, such as updating build processes, package manager configs, etc. Does not trigger a release.
  • build: Changes that affect the build system or external dependencies. May trigger a release.
  • ci: Modifications to continuous integration configuration files and scripts. Does not trigger a release.
  • revert: Reverts a previous commit. It could result in a patch, minor, or major release.
• MR contains database changes including schema/background migrations: not applicable.
• Change contains a breaking change - apply the breaking change label. See "What changes for a running registry" above; I left this to a maintainer.
• Change is considered high risk - apply the label high-risk-change
• I created or linked to an existing issue for every added or updated TODO, BUG, FIXME or OPTIMIZE prefixed comment (there are none).
• Changes cannot be rolled back: not applicable, a revert restores the previous behavior.

Documentation/resources

Code review guidelines

Go Style guidelines

Feature flags

When documentation is required

Documentation workflow

Reviewer checklist

• Ensure the commit and MR tittle are still accurate.
• If the change contains a breaking change, verify the breaking change label.
• If the change is considered high risk, verify the label high-risk-change
• Identify if the change can be rolled back safely. (note: all other reasons for not being able to rollback will be sufficiently captured by major version changes).