Improve instructions around including X-Ray libraries in prompt
Consider if adding clearer instructions about dependencies would help to improve results
The following discussion from !155433 (merged) should be addressed:
-
@jprovaznik started a discussion: (+2 comments) not-blocking: I was wondering if we would get better results with updating the instruction to something like
The list of libraries used by this project is provided in <libs></libs> tags. You can use code from these libraries to generate new code.?🤔 But we can tune this separately as this is not introduced by this MR.
Proposal
Use anthropic tools as well as LangSmith/ELI5 to evaluate X-Ray instruction changes. Document the changes and the rationale behind them. Then apply the new instructions behind a feature flag and monitor acceptance rate.
This issue primarily serves as a POC and trial of the prompt evaluation process. It will help inform us for the next iteration of improving the entire Code Generation prompt in Refine the Code Generation Prompt (#474477 - closed).