feat(sdlc): sdlc indexer task scaffolding

What does this MR do and why?

• Scaffolds the indexing architecture, which include a basic worker pool, listening to nats for various (indexing for now) tasks and routing to the correct handler.
  • The architecture is extensible to multiple types of tasks, meaning it is ready for real-time indexing and code indexing.
• Cleans up legacy listening code

Note 1: All the added parts are not production ready. They're put in place to help guide the development towards a scalable and flexible service.

Note 2: The worker pool is only for development purposes, it does not implement distributed coordination. This will be handled in future parts of #279.

Related Issues

Relates to #279

Testing

Tested locally with Tilt

Performance Analysis

• This merge request does not introduce any performance regression. If a performance regression is expected, explain why.