Real-time updates not working

Summary

Real-time updates are not working correctly, gitlab returns HTTP 304 although changes were made.

In our GitLab instance this affects for example the issue title and the pipeline pages. The build log updates correctly, although I'm not sure whether the mechanism is the same.

What I can see from gitlab-ctl tail is that gitlab-workhorse returns a 304 status code, so this seems to be some kind of caching issue.

This results in not only not updating the contents, but also displaying outdated contents after the page is manually refreshed. After loading the issue page for example, the title is displayed correctly for a split second and then "updates" to the outdated title. After refreshing the page with Ctrl+F5 the contents are displayed correctly.

When waiting approximately 20 minutes after making a change, the page refreshes correctly. So there seems to be some cache which expires after a couple minutes. When disabling the browser cache via the developer console in Chrome, the page refreshes correctly.

Steps to reproduce

Example 1:

• Create an issue
• Edit the issue to update the title
• Go back to the issue page, the old title is displayed
• Refresh with F5 still displays old content
• Refresh with Ctrl+F5 updates the page correctly

Example 2:

• Go to Pipeline page
• Push a new commit which triggers a pipeline
• The pipeline page does not update
• Refresh with F5 still displays old content
• Refresh with Ctrl+F5 updates the page correctly

What is the current bug behavior?

Real-time update not working because GitLab return HTTP status code 304

What is the expected correct behavior?

GitLab should return the update conten with status 200

Relevant logs and/or screenshots

After updating the issue title, gitlab-ctl tail shows the following:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/gitlab/twucher/test/issues/1/rendered_title" for 192.168.25.52 at 2017-06-01 11:12:50 +0200
Started GET "/gitlab/twucher/test/noteable/issue/83/notes" for 192.168.25.52 at 2017-06-01 11:12:50 +0200
Processing by Projects::NotesController#index as JSON
  Parameters: {"namespace_id"=>"twucher", "project_id"=>"test", "target_type"=>"issue", "target_id"=>"83"}

==> /var/log/gitlab/gitlab-workhorse/current <==
2017-06-01_09:12:50.47245 edulis @ - - [2017-06-01 11:12:50.455985847 +0200 CEST] "GET /gitlab/twucher/test/issues/1/rendered_title HTTP/1.1" 304 0 "https://edulis/gitlab/twucher/test/issues/1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 0.016405

==> /var/log/gitlab/nginx/gitlab_access.log <==
127.0.0.1 - - [01/Jun/2017:11:12:50 +0200] "GET /gitlab/twucher/test/issues/1/rendered_title HTTP/1.0" 304 0 "https://edulis/gitlab/twucher/test/issues/1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

In the screenshot you can see that i updated the title to testupdate7 three minutes ago, but the title still reads testupdate6:

Output of checks

Results of GitLab environment info

System information
System:		Debian 9.0
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Git Version:	2.11.1
Sidekiq Version:5.0.0

GitLab information
Version:	9.2.2
Revision:	595b4e9
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://edulis/gitlab
HTTP Clone URL:	https://edulis/gitlab/some-group/some-project.git
SSH Clone URL:	git@edulis:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: redmine

GitLab Shell
Version:	5.0.4
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 5.0.4 ? ... OK (5.0.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
2/2 ... ok
2/3 ... ok
6/9 ... ok
7/12 ... ok
7/13 ... ok
7/15 ... ok
7/16 ... ok
12/18 ... ok
12/20 ... ok
7/22 ... ok
7/23 ... ok
13/25 ... ok
12/28 ... ok
13/35 ... ok
2/36 ... ok
12/38 ... ok
7/42 ... ok
13/43 ... ok
13/44 ... ok
27/46 ... ok
27/47 ... ok
27/48 ... ok
27/49 ... ok
27/50 ... ok
25/51 ... ok
25/52 ... ok
25/53 ... ok
25/54 ... ok
25/55 ... ok
25/56 ... ok
25/57 ... ok
12/58 ... ok
12/59 ... ok
12/61 ... ok
12/62 ... ok
13/63 ... ok
13/67 ... ok
26/68 ... ok
12/69 ... ok
13/70 ... ok
13/71 ... ok
2/72 ... ok
26/73 ... ok
6/74 ... ok
13/76 ... ok
12/77 ... ok
13/78 ... ok
13/79 ... ok
13/80 ... ok
13/81 ... ok
13/82 ... ok
13/83 ... ok
13/84 ... ok
13/85 ... ok
13/86 ... ok
13/87 ... ok
13/88 ... ok
13/89 ... ok
2/90 ... ok
12/91 ... ok
2/92 ... ok
26/93 ... ok
20/94 ... ok
6/95 ... ok
13/96 ... ok
13/97 ... ok
13/98 ... ok
2/100 ... ok
12/101 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ... 
2/2 ... yes
2/3 ... yes
6/9 ... yes
7/12 ... yes
7/13 ... yes
7/15 ... yes
7/16 ... yes
12/18 ... yes
12/20 ... yes
7/22 ... yes
7/23 ... yes
13/25 ... yes
12/28 ... yes
13/35 ... yes
2/36 ... yes
12/38 ... yes
7/42 ... yes
13/43 ... yes
13/44 ... yes
27/46 ... yes
27/47 ... yes
27/48 ... yes
27/49 ... yes
27/50 ... yes
25/51 ... yes
25/52 ... yes
25/53 ... yes
25/54 ... yes
25/55 ... yes
25/56 ... yes
25/57 ... yes
12/58 ... yes
12/59 ... yes
12/61 ... yes
12/62 ... yes
13/63 ... yes
13/67 ... yes
26/68 ... yes
12/69 ... yes
13/70 ... yes
13/71 ... yes
2/72 ... yes
26/73 ... yes
6/74 ... yes
13/76 ... yes
12/77 ... yes
13/78 ... yes
13/79 ... yes
13/80 ... yes
13/81 ... yes
13/82 ... yes
13/83 ... yes
13/84 ... yes
13/85 ... yes
13/86 ... yes
13/87 ... yes
13/88 ... yes
13/89 ... yes
2/90 ... yes
12/91 ... yes
2/92 ... yes
26/93 ... yes
20/94 ... yes
6/95 ... yes
13/96 ... yes
13/97 ... yes
13/98 ... yes
2/100 ... yes
12/101 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.11.1)
Active users: 13

Checking GitLab ... Finished

Possible fixes

I'm not sure where this issue comes from, but I have a few guesses what could be related to this:

• Relative URL configuration. We had several problems in the past regarding relative URL configuration so maybe there is some caching rule not including the relative URL.
• Our GitLab is behind a reverse proxy. Since I can see the requests with gitlab-ctl, I assume the reverse proxy is fine and the caching is not done in the proxy. Nevertheless it might be somehow related. The reverse proxy sets all headers as described in the GitLab installation guide.

As a workaround I would like to test whether disabling caches in gitlab-rails/unicorn helps. Is this easily possible somehow?