fix: availability and data loss fixes (#783)

Phase A of the code quality action plan. Closes #783 (closed).

Four defects that can kill the engine process or lose clone state, one commit each.

1. Token sweep race (internal/srv/ws/token.go)

cleanUpTokens ranged over jwtRegistry without the lock while IssueToken wrote to it. The sweep now snapshots keys under tk.mu, validates lock-free, and deletes the expired set under one hold. ValidateToken still takes the same mutex via isTokenExists, so the sweep never calls it while locked.

2. Webhook registry race and leaks (internal/webhooks/webhooks.go)

Reload swapped hooksRegistry while Run read it. Added sync.RWMutex; Run gets a copy of the hook slice. Response body is closed after every request, non-2xx is logged. Client has a 10s timeout and a bounded transport.

3. No panic recovery in background goroutines (pkg/util/goroutine)

New package with Go (one-shot), Loop (restart with 1s..1m backoff until ctx ends), Run (synchronous, for the cron refresh job). All recover, log the name and stack, never re-panic.

Launch sites and policy:

site policy
whs.Run, RunCleaningUp, removeObservingClones, setReloadListener, CollectUsage (main.go) restart
runAutoDeletion, StartBackgroundCollection (srv/server.go) restart
runIdleCheck, runProtectionLeaseCheck (cloning/base.go) restart
scheduled full refresh (retrieval.go) Run inside the cron job
server.Run, embeddedUI.Run, ResolveUpgradeTarget, triggerWebhook one-shot

Request-scoped goroutines (StartSession, ResetSession, destroyClone) untouched. Explicit lock/unlock pairs in the restarted loop bodies are trivial assignments that cannot panic, left as is.

4. Torn sessions.json destroys every clone (internal/cloning/storage.go, provision/mode_local.go)

Bare os.WriteFile; a crash mid-write left a file that decoded into an empty map, after which stopPoolSessions destroyed every clone dataset. Now: temp file in the same dir, fsync, chmod 0600, rename, directory fsync; temp removed on any error. Decode error is wrapped and Base.Run returns it, so startup stops before cleanupInvalidClones.

stopPoolSessions hardcoded main/r0 for every dataset. Added ListCloneDatasets to FSManager: ZFS parses branch, clone and revision from the dataset name (ListClonesNames now derives from it and drops the "check revision suffix" TODO), LVM maps to the default branch. Both mocks updated.

Also added log.SetOutput/ResetOutput so tests can assert on log lines.

Every task has a -race test that fails before the fix (token sweep and webhook reload verified against the pre-fix code). make test and golangci-lint run are green.

Not in this MR: the read-only-dir save test skips as root; the missing-dir variant covers the no-partial-file guarantee unconditionally.

Merge request reports

Loading
Loading