Code quality plan, MR A: availability and data loss fixes

Phase A of the code quality action plan (https://gitlab.com/postgres-ai/database-lab/-/blob/docs/code-quality-action-plan/docs/plans/20260913-code-quality-assessment-action-plan.md). One MR.

Four defects that can kill the engine process or lose clone state. All confirmed on origin/master 8a42a6ef.

Findings

  • cleanUpTokens ranges over jwtRegistry without the lock (internal/srv/ws/token.go:132); concurrent map write panics the process
  • webhooks.Reload replaces hooksRegistry with no mutex; response body never closed; client has no timeout
  • zero recover() across all background goroutine launch sites
  • sessions.json written with bare os.WriteFile (internal/cloning/storage.go:147); a torn file on restart yields an empty map and stopPoolSessions destroys every clone dataset; branch/revision hardcoded at mode_local.go:756

Tasks

  • 1. Lock the token sweep: snapshot keys under tk.mu, validate lock-free, delete under one hold (isTokenExists takes the same non-reentrant mutex)
  • 2. RWMutex on the webhook registry; defer resp.Body.Close(); client timeout and bounded transport
  • 3. goroutine.Go(name, fn) recover wrapper on long-lived loops only, with a per-site restart-or-one-shot policy
  • 4. Atomic sessions.json: temp file, fsync, rename, directory fsync; Base.Run returns the load error instead of logging it; stopPoolSessions derives branch/revision from the dataset name

Each task gets a -race test that fails before the fix. Full checklist in the plan.