Skip to content

[QA] Enable code_suggestions_in_web_ide_spec.rb QA spec

Description

The QA spec code_suggestions_in_web_ide_spec.rb was added to fast quarantine since it continued to fail on staging. This blocked deployments, resulting in this incident.

TODO: Let's investigate why this was failing and re-enable the test as soon as possible.

Investigation

  • Our first thought was that this was related to the new web_ide_oauth Feature Flag. However, this proved to not be the case (internal Slack thread).
  • This QA spec will fail even if everything is hooked up properly, but the model has no suggestions. Maybe we need to improve the prompt?
  • Upon testing, whenever the code_suggestions/completions endpoint returns with model.name as code-bison@latest, we get no results. code_gecko@latest performs much better and returns results. Maybe there was a change in the AI Gateway that is routing certain requests to the underperforming code-bison model?

Thanks

Thanks @mayra-cabrera && @ddavison for collaborating on this investigation!

Edited by Paul Slaughter