Allow users to opt out of prompt caching for code completion
We would like to implement an opt-out mechanism for any users who do not want to use prompt caching for code completion.
I think the implementation would be something like this:
- Add a top-level namespace setting (in GitLab Rails) to let admins opt out of prompt caching
- This setting should then apply to all groups and projects within the top-level namespace.
- Pass that information along in the headers which we pass to the client when fetching direct_access (see code here). We need to make sure this information is added for indirect connections as well.
- Client/IDE attaches the headers to each request sent to AIGW (this already happens)
- AIGW checks the header and passes the information to Fireworks (see code here)
AI Settings notes
We will also need to work across teams to ensure there's an AI Settings admin option to support the opt out. Prior details on this related to Chat are captured here: &16708 (closed)
Edited by Jordan Janes