Redis cache inconsistency between canary and production

On web-cny-01-sv-gprd, I am seeing odd behavior where the cache does not seem to be consistent between canary and production. I noticed this because my source branch in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23607 claims not to exist:

image

On web-cny-02:

irb(main):018:0> proj = Project.find(13083)        
=> #<Project id:13083 gitlab-org/gitlab-ce>
irb(main):019:0> proj.repository.branch_names.include?('sh-remove-gitlab-shell-include')
=> false
irb(main):020:0> proj.repository.branch_names.count
=> 2226                               

On web-01-sv-grpd:

irb(main):011:0> proj = Project.find(13083)        
=> #<Project id:13083 gitlab-org/gitlab-ce>
irb(main):012:0> proj.repository.branch_names.include?('sh-remove-gitlab-shell-include')
=> true
irb(main):013:0> proj.repository.branch_names.count
=> 2230

I checked that both nodes are using the same /var/opt/gitlab/gitlab-rails/etc/redis.cache.yml, although it's possible the application on canary is ignoring that file or using a different cache key due to the Rails 5 upgrade.

Edited Dec 06, 2018 by Stan Hu
Assignee Loading
Time tracking Loading