Disable Nginx cache for everything but /assets/

My testing show that Nginx responds with 200 instead of 304 to all queries with If-Modified-Since when upstream resource responds with any cache disabling Cache-control value, especially private. It still uses 304 when communicating with workhorse but for some reason 200 and full payload is returned to the user.

Since this is also happening on every request and it also disables If-Modified-Since for the browser requiring transfer of full avatar everytime. Effectively nullifying any possible caching benefits. Because local connection is still made through to Rails, and the whole image is transferred from our DC.

So by disabling caching we can at least rely on the If-Modified-Since mechanism

Update: I checked various settings and checked other endpoints like /emojis I couldn't get Nginx to handle Cache-Control private and no-cache properly in both cases Nginx always forced web browsers to download the whole file instead. Thus I've modified this MR to disable Nginx cache for whole Rails app save for /assets path which uses Cache-Control: public and that is handled correctly

This might seem backwards but to enable caching we have to disable it in NGINX since most of our endpoints outside of /assets doesn't require caching and those that do most often use Cache-Control: private/no-cache which isn't handled by NGINX properly.

FWIW I tried running GDK through Varnish and as far as I've checked the cache handling worked perfectly as it should Out of the box. With appropriate revalidation, long term caching, and handling of Cache-Control: private

Related to gitlab-ce#27936