Skip to content
Snippets Groups Projects

fix: refine troubleshooting script

fix: refine troubleshooting script
mmj-troubleshoot-refine into main
4 unresolved threads
Merged Manoj M J requested to merge mmj-troubleshoot-refine into main
4 unresolved threads

What does this merge request do and why?

For https://gitlab.com/gitlab-com/request-for-help/-/issues/2277, because Support had trouble running the troubleshoot command since the options are confusing.

To resolve this, I've made a few changes:

  • The arguments to the command now directly corresponds to what we show in the UI when creating a self-hosted model.

    • --model-key has been renamed to --api-key
    • --model-name has been renamed to --model-family

  • provider specific checks are made only if the provider is bedrock. This is required because, at times when the user inputs model-identifier as, say, custom_openai/Mixtral-8x7B-Instruct-v0.1, platform will be set to custom_openai, and the script would just error out with ValueError: Provider Custom_openai is not a supported provider.. This is an unwarranted check to perform anyway.

After this change, the usage of the command looks like:

Examples:

For a claude_3 model running on AWS Bedrock:

poetry run troubleshoot \
  --model-family=claude_3 \
  --model-identifier=bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0

For a mixtral model running on vLLM:

poetry run troubleshoot \
  --model-family=mixtral \
  --model-identifier=custom_openai/Mixtral-8x7B-Instruct-v0.1 \
  --api-key=your-api-key \
  --model-endpoint=http://<your-model-endpoint>/v1

Complementary doc changes added at: gitlab-org/gitlab!179973 (merged)

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Manoj M J

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Tan Le
  • Tan Le
  • Tan Le
  • Tan Le requested review from @tle_gitlab

    requested review from @tle_gitlab

  • Tan Le added maintenancerefactor workflowin review labels

    added maintenancerefactor workflowin review labels

  • Manoj M J added 1 commit

    added 1 commit

    Compare with previous version

  • Manoj M J reset approvals from @eduardobonet by pushing to the branch

    reset approvals from @eduardobonet by pushing to the branch

  • Manoj M J added 1 commit

    added 1 commit

    Compare with previous version

  • Manoj M J
    Manoj M J @manojmj ·
    Author Developer

    @tle_gitlab thank you for the review, I have addressed your comments. Please take a look :ping_pong: :slight_smile:

  • Manoj M J removed review request for @eduardobonet

    removed review request for @eduardobonet

  • Tan Le
    Tan Le @tle_gitlab started a thread on the diff
    • ai_gateway/scripts/troubleshoot_selfhosted_installation.py
    7 7 import boto3
    8 8 import requests
    9 9
    10 # Constants
  • Tan Le
    Tan Le @tle_gitlab started a thread on the diff
    • ai_gateway/scripts/troubleshoot_selfhosted_installation.py
    213 215
    214 parser = argparse.ArgumentParser(description="Test AI Gateway and model access")
    216 parser = argparse.ArgumentParser(
    217 description="Test AI Gateway and model access",
    218 formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    219 )
    215 220 parser.add_argument(
    216 "--endpoint", default="localhost:5052", help="AI Gateway endpoint"
    221 "--endpoint",
    222 default="localhost:5052",
    223 help="AI Gateway endpoint.",
    217 224 )
    218 225 parser.add_argument(
    219 "--model-name", required=False, help="Name of the model to test"
    226 "--model-family",
    227 required=False,
  • Tan Le
    Tan Le @tle_gitlab started a thread on the diff
    • ai_gateway/scripts/troubleshoot_selfhosted_installation.py
    229 help="Family of the model to test.",
    220 230 )
    221 231 parser.add_argument(
    222 232 "--model-endpoint",
    223 233 required=False,
    224 234 default="http://localhost:4000",
    225 help="Endpoint of the model. Example: http://localhost:4000. "
    235 help="Endpoint of the model."
    226 236 "When using a model from an online provider like Bedrock, "
    227 "this can be left empty.",
    237 "this can be left empty. When using a model hosted via vLLM, the /v1 suffix is required.",
    228 238 )
    229 239 parser.add_argument(
    230 "--model-identifier", required=False, help="Identifier of the model"
    240 "--model-identifier",
    241 required=False,
  • Tan Le
    Tan Le @tle_gitlab started a thread on the diff
    • ai_gateway/scripts/troubleshoot_selfhosted_installation.py
    215 220 parser.add_argument(
    216 "--endpoint", default="localhost:5052", help="AI Gateway endpoint"
    221 "--endpoint",
    222 default="localhost:5052",
    223 help="AI Gateway endpoint.",
    217 224 )
    218 225 parser.add_argument(
    219 "--model-name", required=False, help="Name of the model to test"
    226 "--model-family",
    227 required=False,
    228 choices=SUPPORTED_MODEL_FAMILIES,
    229 help="Family of the model to test.",
    220 230 )
    221 231 parser.add_argument(
    222 232 "--model-endpoint",
    223 233 required=False,
  • Tan Le approved this merge request

    approved this merge request

  • Manoj M J added 1 commit

    added 1 commit

    • 623072b2 - fix: refine troubleshooting script

    Compare with previous version

  • Tan Le started a merge train

    started a merge train

  • Tan Le merged

    merged

  • Tan Le mentioned in commit c2385d45

    mentioned in commit c2385d45

    • Please register or sign in to reply