Allow duo_core add-on for generate_embeddings_codebase

What does this MR do and why?

In feat: Add unit primitive for generate_embedding... (!146 - merged), we introduced the generate_embeddings_codebase unit primitive for the Semantic Code Search, with the supported add-ons being duo_core and duo_enterprise. However, the tier availability for this feature had changed since then, and we need to support it for duo_core.

Diff

main...588259-allow-duocore-generate_embeddings_codebase

References

• Semantic Code Search epic: Semantic search: chat with your codebase (gitlab-org#16910)
• Semantic Code Search doc: https://docs.gitlab.com/user/gitlab_duo/semantic_code_search/
• Related bug/issue submitted by Duo Core user: Semantic code (ActiveContext Code) search index... (gitlab-org/gitlab#588259 - closed)

How to set up and validate locally

Check the update in Rails.

1. Set GITLAB_SIMULATE_SAAS=0 in your GDK/Rails environment variable

2. Update your add-on to only have duo_core

   GITLAB_SIMULATE_SAAS=0 bundle exec 'rake gitlab:duo:setup[duo_core]'

3. Update the the gitlab-cloud-connector gem to refer to the local path (see instructions)

4. Generate a JWT token

   token = Gitlab::AiGateway.cloud_connector_token('generate_embeddings_codebase', User.first)

5. Decode the token

   decoded_hash = JWT.decode(token, nil, false)

6. Verify that the scopes include the generate_embeddings_codebase unit primitive

   decoded_hash.first['scopes'].include?('generate_embeddings_codebase')
   => true

   Additional verification: without the update, this should return false.