As a developer seeking quick solutions, I want to be able to ask Duo Chat questions about my terminal output and get immediate help without leaving my workflow.
Job Story
When I encounter a confusing error message in my terminal, I want to be able to quickly share it with Duo Chat and receive a clear explanation or potential solution so that I can unblock myself and continue coding efficiently.
Why is this important?
Immediate, context-aware assistance. Users receive immediate, context-aware aid in their workflow, saving time and reducing frustration.
Always-available resource. Duo Chat can remain an always-available resource that explains concepts and provides guidance.
Enhanced user experience. Integrating Duo Chat into the terminal can significantly improve the dev experience by providing a seamless and intuitive way to access help.
Proposal
User selects terminal output/snippet
On right click, has menu option to Explain this output
On the back-end we make sure that we strip the first part of the response that starts with "This is not actually code..."
@tvanderhelm Here's the enabling chat in the terminal overview. I think the menu text we show has room for improvement. My proposal is to reuse the current state and have "Explain selected code," but we may need something more terminal-specific.
Having already been familiar with this client and backend code, I would caution against using the existing /explain , since it specifically uses prompts that are for explaining code and not for terminal output (prompt reference).
TLDR; you will get bad (or confusing) results.
Terminal Output
To demonstrate this, I played around with the Duo Chat API.
Here is a simple NPM error that a user might copy into Duo Chat:
We will use the above error as input for selectedText (which is what /explain selected code uses under the hood).
Using Existing Tool
In the backend, these slash commands are considered "tools" (you can read more about that here, thanks @jessieay for the great docs).
Here is the response from the GraphQL API. See the response on the right. This is using GitLab production:
Notice how the LLM is "confused" at first. This is because the prompt is instructing the model to predict tokens for code:
Utils::Prompt.as_user(<<~PROMPT.chomp %<file_content>s Here is the code user selected: <selected_code> %<selected_text>s </selected_code> %<input>s Any code blocks in response should be formatted in markdown. PROMPT)].freeze
(In my opinion) I'm not sure we'd want to put this out to users if this response is easily reproducible. Here's another example (using the same terminal output):
Creating a new Duo Chat Tool
To preface, this is just a demo. What we'd like the UX to be like is up to Product/UX. As a demonstration, I created a new slash command called /explain_terminal.
Here is the tool's corresponding MR (you can see the full prompt there): gitlab!159160 (closed)
The prompt now looks something like this:
PROMPT_TEMPLATE=[Utils::Prompt.as_system(<<~PROMPT You are a Terminal output expert. %<language_info>s PROMPT),Utils::Prompt.as_user(<<~PROMPT.chomp %<file_content>s Here is terminal output the user selected: <selected_terminal> %<selected_text>s </selected_terminal> %<input>s Your response response should be formatted in markdown (but don't mention markdown in the response). PROMPT)].freeze
I put in the same exact terminal output as shown in the Terminal Output section. Below is the result:
1st request:
2nd request: To me, this response quality seems much better.
What to Do
I suggest syncing with some Duo Chat folks about their thoughts on creating a new chat tool. It was relatively simple to add, but there may be gotchas I am missing.
Adding a slash command on the UI is relatively easy, you just add the command here.
If UX/Product doesn't wish to add a new slash command, you may be able to extend the GQL Api so that it takes a new param terminalOutput, and updates the main /explain prompt by adding in that terminal context. I advise against this, though, because making prompts larger with various instructions might not lead to optimal results.
I agree. The first sentence from chat that "the user has not actually selected any code in this case.." does not set us up for success, even though chat does explain the terminal output.
Let's do this:
@tlinz@katiemacoy We want to expand Duo chat capabilities (unlock menu action for /explain) in the terminal, but using /explain forces chat to think of a response as a snippet of code, making the structure of the response a poor user experience. Would you object to us adding a new slash command tool per the docs above for terminal-specific requests?
This will be specific to the IDEs, unless there's a practical use case outside of the IDE?
@tvanderhelm Anything we would want to add from an interaction standpoint outside of these two:
On right click, Duo menu action shows Explain this output or Explain
In the chat menu, a new slash command that is /terminal or something similar.
@andr3 since we're dealing with the terminal, do we want AppSec to weigh in once we have an implementation path forward?
I think explaining terminal output makes sense. I would lean on the actual UI menu more so than a slash command. This is more discoverable.
I do not think we need a new slash command for explaining the terminal output. We are trying to keep the slash command list curated to the most valuable interactions. More info in this WIP MR. I think we could leverage the exist /explain for both the terminal and code. Also for the right click menu.
If we include the context of the terminal output, we need to figure out what the ideal UX is when the user uses /explain without any code or anything in the terminal selected.
@katiemacoy Gotcha. Yes, the slash command menu can get lengthy. For the scope of this issue, on the UX side, @tvanderhelm LMK if you agree: we'll add the context menu setting for Explain in the terminal, without the additional slash command.
@dashaadu I agree with @katiemacoy that adding a terminal-specific slash command likely isn't the road we want to go down.
My hypothesis is that users will see any explain actions as being generic across contexts (files, terminal, etc.). So training users that /Explain is only relevant for code or adding slash commands for each context (/Explain_terminal) is going to be a losing battle.
Just to make sure I understand, the proposal is to use the context menu to provide the command for explaining the terminal output? If so, are we replacing the existing Explain this code with Explain this output and using a terminal specific prompt on the back end?
Also, out of scope but I think using @ mentions to target context could be a nice way to solve this workflow in a more flexible way and lower the temptation to keep adding slash commands.
For example, as a user I could just immediately target the specific context I want in Duo Chat instead of moving between the terminal/chat using copy and paste or highlighting/right clicking to get context menu open. Something like: how do I fix this error @terminal.
This also would fit with the proposal in &14407 because we could default to the open code file if a context isn't specified with @.
@tvanderhelm thanks for your thoughts - I wanted to shed some light on the @context discussion. Since the chat can be used both in the IDE and GitLab, I'm a little bit wary of using @ because this symbol is how you tag users in GitLab. The plan is that eventually the user can attach gitlab specific references to the chat, and that will use the same UI as our quick actions, like this:
Avoiding @ to mean something else is a pretty loosely held opinion - I think there is also an argument that the IDE chat and GitLab chat could diverge. I am only afraid of muddying the user's understand of @ when we implement GitLab specific references. I would like to keep the chat in GitLab and the IDE as similar as possible so the user does not need to learn 2 tools.
Maybe a middle ground would be adding a UI control to add context.
If so, are we replacing the existing Explain this code with Explain this output and using a terminal specific prompt on the back end?
As a first iteration, I'm thinking of:
User selects terminal output
On right click, has menu option to Explain this output
On the back-end we make sure that we strip the first part of the response that starts with "This is not actually code..."
The unknown for me is how much more users would want to interact with the terminal past this. If we get positive signals, adding a UI control would be the second iteration. WDYT?
I would caution against using the existing /explain , since it specifically uses prompts that are for explaining code and not for terminal output (prompt reference).
On the back-end we make sure that we strip the first part of the response that starts with "This is not actually code..."
@andr3 from what I understand, we will need rework our current backend implementation for /explain a bit to accommodate both code and terminal text before we can move forward with this issue. Do we have someone on the backend who can help out here?
@dashaadu@tvanderhelm could you please clarify here what we're aiming at for first iteration?
What do you mean by backend? The Duo Chat backend? That would be handled by the Duo Chat team, correct? If so, we'd need to block this issue until they've implemented this, which would surely slip this from 17.3
If that's the case, we should get an issue prepared documenting what we need them to work on and see when they could get to it. I can then reach out to their EM to explain the need in terms of timing. Thanks!
If it's not the case and we want to handle it from the Extension side only, then we're probably missing some bits... how should we adapt the request we're making to get the proper response based on the nature of the content?
Whenever you have a chance, can you braindump what you had already looked into on this topic? Thanks in advance. Once you do, please unassign yourself. ;)
The main caveat is that VSCode doesn't yet have an API for accessing the terminal's selected text. There's an open issue for this in the VSCode repository.
Workaround:
As a workaround, we can copy the selected terminal text to the user's clipboard using the workbench.action.terminal.copySelection VSCode command, then retrieve the text with vscode.env.clipboard.readText(). However, with this workaround we'll need to restore the user's clipboard to its original state afterward, so they don't lose their previous clipboard contents.
POC
I've started on a POC for this feature using the workaround mentioned above, it should give you a rough idea of a direction we can take.
Backend
We do not yet have a command on the backend for explaining terminal output. @marvingay is working on this over at gitlab#477749 (closed).
If you have any questions about anything, please reach out
Thanks @elwyn-gitlab! Let me know if we don't have a clear path to update the back-end and make it terminal specific. We can go to a more boring solution to start, where we send the highlighted output to Duo and let the user type in the prompt, but would be great if we can do it in a single step.
Please provide a quick summary of the current status (one sentence)
Current status: The current proposed approach is to re-use the existing /explain command. -- Currently determining the circumstances around the possibility of LLM answer degradation if the prompt is modified and how to correctly assess.
How confident are you that this will make it to the current milestone?
Not confident
Slightly confident
Very confident
Are there any opportunities to further break the issue or merge request into smaller pieces (if applicable)?
Yes
No
Were expectations met from a previous update? If not, please explain why.
Yes
No, ___
Are the related issues up to date? Please link any missing issues in the epic that could be linked to this issue
Caught up with @marvingay this week and we've been through the options, just waiting on solidifed backend + some testing once we're integrated before I can proceed into review.
No changes since last update, my MR is still ready for integration + testing with the backend.
I have updated this to workflowblocked - I misunderstood that the backend issue was only aiming to deliver the spike outcome this milestone. So this issue is blocked until we deliver an implementation based on that spike (new issue being discussed here, I will set this as blocked by that when it is opened).
As such, it is unlikely this one will be delivered in %17.6
I will do some further investigation on this in the meantime regarding Angelo's suggestion of using the context architecture as the mechanism for implementing 1. MVC: Basic functionality as set out by Dasha in the thread above. I don't think this will impact the backend for this first iteration too much, it will just change where on the chat message object the terminal output content is stored.