Add score to Semantic Code Search MCP tool results

What does this MR do and why?

Related to #579850 (closed)

FF: #586946

The result is already ranked, by the semantic distance from query string. So that the first hit is the closest to the query string, and so on. Although we know this, there's no explanation to LLM. Also, limit is somewhat arbitrary, for example the hits after the 10th result are irrelevant.

after the implementation, the response which is feed to LLM will be:

  1. app/services/auth/jwt_service.rb (score: 0.9523)                                           
     def verify_token(token)                                                                    
       # JWT verification logic...                                                              
                                                                                                
  2. lib/middleware/authentication.rb (score: 0.8741)                                           
     class AuthenticationMiddleware                                                             
       # middleware implementation...  

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. set the mcp client:

!205297 (comment 2756113040)

2. check the description and send the request:

{
  "content": [
    {
      "type": "text",
      "text": "1. duo_workflow_service/workflows/abstract_workflow.py\n   #  tgao xx3 \n2. tests/duo_workflow_service/components/human_approval/test_tools_approval.py\n               ]"
    }
  ],
  "structuredContent": {
    "items": [
      {
        "project_id": 1000000,
        "path": "duo_workflow_service/workflows/abstract_workflow.py",
        "content": "#  tgao xx3 ,
        "name": "abstract_workflow.py",
        "language": "python",
        "blob_id": "057040676a7cd12f9a56c06a59278c5a07205e4c",
        "start_byte": 11721,
        "length": 1000,
        "start_line": 291
      },
      {
        "project_id": 1000000,
        "path": "tests/duo_workflow_service/components/human_approval/test_tools_approval.py",
        "content": "            ]",
        "name": "test_tools_approval.py",
        "language": "python",
        "blob_id": "96985e997aacef1469d89769173b55222fa10507",
        "start_byte": 13973,
        "length": 13,
        "start_line": 390
      }
    ],
    "metadata": {
      "count": 2,
      "has_more": false
    }
  },
  "isError": false
}

3. enable the ff:

[2] pry(main)> ::Feature.enable(:post_process_semantic_code_search_add_score)

4. check the description and send the request:

{
  "content": [
    {
      "type": "text",
      "text": "1. tests/api/v2/test_v2_code.py (score: 0.7998)\n#  tgao xx3\n2. tests/duo_workflow_service/components/human_approval/test_tools_approval.py (score: 0.7981)\n               ]"
    }
  ],
  "structuredContent": {
    "items": [
      {
        "project_id": 1000000,
        "path": "tests/api/v2/test_v2_code.py",
        "content": "#  tgao xx3",
        "name": "test_v2_code.py",
        "language": "python",
        "_score": 0.7998023,
        "blob_id": "f8be989993d07f0e7d472e32db1f6f7b0bc00981",
        "start_byte": 4445,
        "length": 27,
        "start_line": 141
      },
      {
        "project_id": 1000000,
        "path": "tests/duo_workflow_service/components/human_approval/test_tools_approval.py",
        "content": "            ]",
        "name": "test_tools_approval.py",
        "language": "python",
        "_score": 0.7981448,
        "blob_id": "96985e997aacef1469d89769173b55222fa10507",
        "start_byte": 13973,
        "length": 13,
        "start_line": 390
      }
    ],
    "metadata": {
      "count": 2,
      "has_more": false
    }
  },
  "isError": false
}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.