Improve performance of /api/v4/suggestions/batch_apply
Sid mentioned in the CEO channel on www-gitlab-com::
Make two suggestions. Batch them. Press apply button, but doesn't show that anything is happening. (there is an issue for some messaging bugs but I'm not sure that is it) Reload and hit problem 1. again. It seems the suggestions did apply.
I believe this was happening on gitlab-com/www-gitlab-com!64326 (merged). From https://log.gprd.gitlab.net/goto/2307c68bac23b86e21cff5d51d15c54e:
For some reason, this request sat in the Workhorse queue for 15 seconds, and then it took 20 seconds to process.
If we drill down to the first request via correlation_id
(https://log.gprd.gitlab.net/goto/cf8d6dadc55a18297bd41944c53f0a74), we see UserCommitFiles
RPC taking most of the time:
This could be a function of the load on the file server, but file-cny-01
seemed under 20% CPU utilization (https://dashboards.gitlab.net/d/bd2Kl9Imk/host-stats-old-prometheus?from=1601935865456&orgId=1&to=1601936188252&var-environment=gprd&var-node=file-cny-01-stor-gprd.c.gitlab-production.internal&var-promethus=prometheus-01-inf-gprd).
Indeed, if we look over the course of the week, applying suggestions consistently took 20 s for Sid (https://log.gprd.gitlab.net/goto/b08b04153e66ddd6c433c5480e90d55b):
/cc: @danielgruesso