Some browsers are getting server responses from GitLab proper instead of Pages
Summary
User visits a (previously working) Pages page and receives a GitLab 404 response, not a GitLab Pages 404 response.
Steps to reproduce
I've been unable to narrow down exactly where the problem lies. It seems to be dependent on the browser, and iOS Safari seems to be the only one that reliably fails.
Using GitLab 12.1.0 CE with wildcard DNS and TLS setup for Pages, with access control enabled in config and turned on for each project. While the user in question was unable to break out of this problem in Chrome or Safari on OS X, Android Chrome, or iOS Safari, I was only able to reproduce eventually on iOS Safari (two other users were unable to reproduce using Chrome on OS X or a Chromebook). The user facing problems cleared all data for every browser on each device before attempting to access a Pages page.
- Access a URL that should be served by Pages
- Encounter a browser SSL warning page, which is the first sign something is wrong. This comes up because while the browser is attempting to access the Pages domain, the actual response comes from GitLab, which in our case uses a different certificate than the wildcard one used for all Pages subdomains. This base GitLab cert doesn't apply to the Pages subdomain, so the browser considers it an error.
- After accepting the risks per browser, proceed; the redirect chain then lands on
/users/sign_in
but on the Pages domain. This actually works because it's getting responses from GitLab. Log in successfully. - Get an HTTP 404 from GitLab because the page requested isn't recognized by GitLab.
I additionally removed the user's rows from oauth_access_grants
and oauth_access_tokens
under the assumption a successful authorization would generate new rows. There still aren't any new rows for them after all of this (but I wouldn't expect them since so far all they've done is login to GitLab from the wrong domain).
An upgrade to 12.1.6 doesn't make any difference.
Example Project
Probably not pertinent, they were unable to access any Pages projects we publish.
What is the current bug behavior?
It sure looks like GitLab is responding instead of passing things on to GitLab Pages despite the domain being requested.
What is the expected correct behavior?
They should see the Pages page or go through the authorization process before landing on the Pages page in the end.
Relevant logs and/or screenshots
From /var/log/gitlab/nginx/gitlab_access.log
, GitLab serving the Pages page:
<IPv6-address> - - [15/Aug/2019:00:43:23 +0000] "GET <pages-page> HTTP/2.0" 404 6301 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36"
From /var/log/gitlab/gitlab-pages/current
, a different user:
{"duration":0.049946554,"host":"[pages-domain]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.0","referer":"","remoteAddr":"127.0.0.1:33716","status":200,"system":"http","time":"2019-08-15T13:06:57Z","uri":"<pages-page>","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36","written":605}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 12.1.6 Revision: 4016bcac51d Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.12 URL: https://[scrubbed] HTTP Clone URL: https://[scrubbed]/some-group/some-project.git SSH Clone URL: git@[scrubbed]:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 9.3.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/1 ... yes 3/2 ... yes 3/3 ... yes 3/4 ... yes 3/5 ... yes 3/6 ... yes 3/7 ... yes 3/8 ... yes 3/9 ... yes 3/10 ... yes 3/11 ... yes 3/12 ... yes 3/13 ... yes 3/14 ... yes 3/15 ... yes 3/16 ... yes 3/17 ... yes 3/18 ... yes 3/19 ... yes 3/20 ... yes 3/21 ... yes 3/22 ... yes 3/23 ... yes 3/24 ... yes 3/25 ... yes 3/26 ... yes 3/27 ... yes 3/28 ... yes 3/29 ... yes 3/30 ... yes 3/31 ... yes 3/32 ... yes 3/33 ... yes 3/34 ... yes 3/35 ... yes 3/36 ... yes 3/37 ... yes 3/38 ... yes 3/39 ... yes 3/40 ... yes 3/41 ... yes 3/42 ... yes 3/43 ... yes 3/44 ... yes 3/45 ... yes 3/46 ... yes 3/47 ... yes 3/48 ... yes 3/49 ... yes 3/50 ... yes 3/51 ... yes 3/52 ... yes 3/53 ... yes 3/54 ... yes 3/55 ... yes 3/56 ... yes 3/57 ... yes 4/58 ... yes 4/60 ... yes 4/61 ... yes 4/62 ... yes 4/63 ... yes 4/64 ... yes 4/65 ... yes 4/66 ... yes 4/67 ... yes 4/68 ... yes 4/69 ... yes 4/70 ... yes 4/71 ... yes 4/72 ... yes 4/73 ... yes 4/74 ... yes 4/75 ... yes 4/76 ... yes 4/77 ... yes 4/78 ... yes 4/79 ... yes 4/80 ... yes 4/81 ... yes 4/82 ... yes 4/83 ... yes 4/84 ... yes 4/85 ... yes 4/86 ... yes 4/87 ... yes 4/88 ... yes 4/89 ... yes 4/90 ... yes 4/91 ... yes 4/92 ... yes 4/93 ... yes 4/94 ... yes 4/95 ... yes 4/96 ... yes 4/97 ... yes 4/98 ... yes 4/99 ... yes 4/100 ... yes 4/102 ... yes 4/103 ... yes 4/104 ... yes 4/105 ... yes 4/106 ... yes 4/107 ... yes 4/108 ... yes 4/109 ... yes 4/110 ... yes 4/111 ... yes 4/112 ... yes 4/114 ... yes 4/115 ... yes 4/118 ... yes 4/119 ... yes 4/120 ... yes 4/121 ... yes 4/122 ... yes 4/123 ... yes 4/124 ... yes 4/125 ... yes 4/126 ... yes 4/127 ... yes 4/128 ... yes 4/129 ... yes 4/130 ... yes 4/131 ... yes 4/132 ... yes 4/133 ... yes 4/134 ... yes 4/135 ... yes 4/136 ... yes 4/137 ... yes 4/138 ... yes 4/139 ... yes 4/140 ... yes 4/141 ... yes 4/142 ... yes 4/143 ... yes 4/144 ... yes 4/145 ... yes 4/146 ... yes 4/147 ... yes 4/148 ... yes 4/149 ... yes 4/150 ... yes 4/151 ... yes 4/152 ... yes 4/153 ... yes 4/154 ... yes 4/155 ... yes 4/156 ... yes 4/157 ... yes 4/158 ... yes 4/159 ... yes 4/160 ... yes 4/162 ... yes 4/163 ... yes 4/164 ... yes 4/165 ... yes 4/166 ... yes 4/167 ... yes 4/168 ... yes 4/169 ... yes 4/170 ... yes 4/171 ... yes 4/172 ... yes 3/173 ... yes 3/174 ... yes 4/175 ... yes 4/176 ... yes 4/177 ... yes 4/178 ... yes 3/179 ... yes 3/180 ... yes 3/181 ... yes 3/182 ... yes 4/183 ... yes 3/184 ... yes 3/185 ... yes 3/186 ... yes 4/187 ... yes 3/188 ... yes 3/189 ... yes 3/190 ... yes 12/191 ... yes 4/192 ... yes 11/193 ... yes 4/194 ... yes 4/195 ... yes 4/196 ... yes 21/197 ... yes 4/198 ... yes 4/199 ... yes 3/201 ... yes 3/202 ... yes 4/204 ... yes 4/205 ... yes 4/206 ... yes 3/207 ... yes 3/208 ... yes 3/209 ... yes 3/210 ... yes 4/211 ... yes 17/212 ... yes 4/214 ... yes 4/215 ... yes 4/216 ... yes 4/217 ... yes 4/218 ... yes 4/219 ... yes 4/220 ... yes 4/221 ... yes 4/222 ... yes 4/223 ... yes 4/224 ... yes 4/225 ... yes 4/227 ... yes 4/228 ... yes 4/229 ... yes 4/230 ... yes 3/231 ... yes 4/232 ... yes 4/233 ... yes 3/234 ... yes 3/235 ... yes 21/236 ... yes 21/237 ... yes 21/238 ... yes 3/239 ... yes 4/242 ... yes 3/243 ... yes 21/244 ... yes 4/245 ... yes 4/246 ... yes 3/247 ... yes 21/248 ... yes 21/250 ... yes 9/251 ... yes 3/252 ... yes 3/253 ... yes 21/254 ... yes 21/255 ... yes 21/257 ... yes 21/258 ... yes 3/259 ... yes 3/260 ... yes 21/262 ... yes 21/263 ... yes 21/264 ... yes 7/265 ... yes 21/266 ... yes 4/267 ... yes 4/268 ... yes 3/269 ... yes 3/270 ... yes 16/271 ... yes 21/272 ... yes 4/273 ... yes 21/274 ... yes 4/275 ... yes 4/276 ... yes 4/277 ... yes 21/278 ... yes 4/279 ... yes 4/280 ... yes 21/281 ... yes 4/282 ... yes 4/283 ... yes 4/284 ... yes 8/285 ... yes 21/286 ... yes 21/287 ... yes 4/288 ... yes 4/289 ... yes 17/290 ... yes 17/296 ... yes 4/297 ... yes 4/298 ... yes 17/299 ... yes 4/302 ... yes 4/303 ... yes 4/304 ... yes 4/305 ... yes 4/306 ... yes 21/307 ... yes 4/308 ... yes 21/309 ... yes 21/313 ... yes 4/314 ... yes 4/315 ... yes 4/316 ... yes 4/317 ... yes 4/318 ... yes 4/319 ... yes 4/320 ... yes 17/321 ... yes 21/323 ... yes 4/324 ... yes 3/325 ... yes 3/326 ... yes 17/327 ... yes 21/328 ... yes 4/329 ... yes 17/330 ... yes 4/331 ... yes 4/332 ... yes 4/333 ... yes 4/334 ... yes 4/335 ... yes 21/336 ... yes 4/337 ... yes 4/342 ... yes 21/343 ... yes 21/346 ... yes 21/347 ... yes 4/348 ... yes 21/349 ... yes 4/350 ... yes 21/351 ... yes 4/352 ... yes 21/353 ... yes 21/354 ... yes 21/355 ... yes 4/356 ... yes 4/357 ... yes 4/358 ... yes 21/359 ... yes 4/360 ... yes 4/361 ... yes 3/362 ... yes 17/363 ... yes 4/364 ... yes 4/365 ... yes 17/366 ... yes 33/367 ... yes 17/368 ... yes 33/369 ... yes 17/370 ... yes 4/371 ... yes 4/372 ... yes 4/373 ... yes 31/374 ... yes 31/375 ... yes 31/376 ... yes 3/377 ... yes 1/378 ... yes 4/379 ... yes 21/380 ... yes 34/381 ... yes 33/382 ... yes 1/383 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.21.0 ? ... yes (2.21.0) Git user has default SSH configuration? ... yes Active users: ... 24
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished