Implement local agent registry

What does this merge request do and why?

These changes generalize the changes from !720 (merged) to provide a generic agent registry interface that we can use to retrieve agents for any completion request

Part of gitlab-org/gitlab#432594 (closed)

Based on the proposal from gitlab-org/gitlab#432594 (comment 1877391883)

How to set up and validate locally

*(validation steps taken from !720 (merged), since they apply here as well)

1. Set the env variable to get access to the new agent endpoint:

   AIGW_DUOCHAT_EXPERIMENTAL=true

2. Request with the empty scratchpad:

   curl -X 'POST' \
      'http://localhost:5052/v2/chat/agent' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
          "prompt": "What'\''s the title of this issue?",
          "options": {
            "chat_history": "",
            "agent_scratchpad": {
              "agent_type": "react",
                "steps": []
            }
          }
         }'

3. Request with the existing scratchpad:

   curl -X 'POST' \
    'http://localhost:5052/v2/chat/agent' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
         "prompt": "What'\''s the title of this issue?",
         "options": {
            "chat_history": "",
            "agent_scratchpad": {
               "agent_type": "react",
               "steps": [
                  {
                     "thought": "To get the title and other details of this issue, I need to use the \"issue_reader\" tool.",
                     "tool": "issue_reader",
                     "tool_input": "What'\''s the title of this issue?",
                     "observation": "here is the issue about implementing a PoC for ...."
                  }
               ]
            }
         }
      }'

4. Request with the existing chat history (either string or a list of strings):

   curl -X 'POST' \
     'http://localhost:5052/v2/chat/agent' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
           "prompt": "What'\''s your name?",
           "options": {
          "chat_history": ["User: What'\''s your name?", "AI: My name is Skynet."],
            "agent_scratchpad": {
              "agent_type": "react",
              "steps": []
            }
           }
         }'

5. Request with the pre-defined resource (issue or epic):

   curl -X 'POST' \
     'http://localhost:5052/v2/chat/agent' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
           "prompt": "What'\''s the title of this issue?",
           "options": {
            "chat_history": "",
            "agent_scratchpad": {
              "agent_type": "react",
              "steps": []
            },
            "context": {
              "type": "issue",
              "content": "title: PoC ReAct, description: some description"
            }
           }
         }'

Merge request checklist

• Tests added for new functionality. If not, please raise an issue to follow up.
• Documentation added/updated, if needed.