[sdlc] Implement job distribution using NATS JetBrains and KV

Problem to Solve

Set up the NATS JetStream and KV infrastructure for managing indexing jobs. Create separate subjects for initial and incremental indexing to enable different worker pools and priorities.

Setup worker pools for different indexing operations (initial, indexing) and create locking mechanism to avoid workers processing the same namespace at the same time.

Proposed Solution

• Create gkg_idnexing_jobs stream

• Create subjects: gkg_indexing_jobs.initial, gkg_indexing_jobs.incremental

• Use NATS KV bucket for indexing locks (example

  /gkg-indexer/indexing/{namespace_id}/lock)

  • Implement lock acquisition with TTL (1 hour default)
  • Store lock metadata: worker_id, started_at
  • Implement lock release on job completion
  • Handle lock contention gracefully (skip to next job)
  • Add metrics for lock acquisition success/failure

• Create consumer groups for worker scaling based on a configuration

• Add metrics around workers to give insight into their availability