Merge request polling is expensive, does not cache adequately, should slow down on background tabs

Each open tab showing a merge request will poll GitLab.com at a high rate, generating expensive uncached requests.

For example, I opened a MR that has not seen activity for several weeks and monitoring the amount of polling traffic the tab generates:

https://log.gitlab.net/goto/b5d9cad45570e91484e00cd8901efa5d

(Some of this time, my laptop was closed)

This is what I found:

• During the time that my laptop was open, the browser tab made ~5 requests per minute.
• The tab was in the background for almost the entire duration of the test. Backgrounding a tab does not appear to slow the polling rate. It should. This would be good for the GitLab.com service and also for users laptop power consumption.
• Not all of these are conditional requests (ie, using ETags or If-Modified-Since), The conditional request polling architect does not appear to being used. Documentation here: https://docs.gitlab.com/ee/development/polling.html
• Other conditional requests still appear which generate a cache hit still appear to be relatively expensive (hitting Gitaly for example)

This all adds up to a lot of traffic generated by polling, most of it on unchanged resources.

• Individual users are generating over 1600 MR polling requests per hour, resulting in more than 8000 gitaly calls per user, just on polling merge requests.
  • Thread: https://gitlab.slack.com/archives/CB3LSMEJV/p1565716333252800
  • ELK report: https://log.gitlab.net/goto/e3322fcccf4f53b74830fc08251da43b
• In total, MR polling is generating about 70,000 gitaly requests per minute at peak. Source https://log.gitlab.net/goto/c73cf894888f51e84abe5bf69833106a
• In total, MR polling is generating about 40 seconds of database requests per second at peak. During today's incident gitlab-com/gl-infra/production#1052 (closed), merge request polling was consuming 250 seconds of database requests per second. It's not yet entirely clear whether this was a cause of the incident or mearly a side-effect. Source https://log.gitlab.net/goto/82e7ca191c0d857e110804e14bf88a03

Gitaly calls generated by merge request polling endpoints:

Database queries generated by merge request polling endpoints:

What can be done?

1. Slowing polling on background tabs is both good for our servers and courteous to our user's laptop battery lifetimes. It would be easy to implement.
2. Consider using the conditional request polling architecture: https://docs.gitlab.com/ee/development/polling.html (or, if there are reasons why it cannot be used in these cases, address those technical failings)