Skip to content

Enable pages_serve_with_zip_file_protocol by default

Jaime Martinez requested to merge 321677-enable-pages-zip-file-protocol-by-default into master

GitLab Pages supports serving content from Zip archives located on disk gitlab-pages#485 (closed). This flag enables the internal API to return a zip source with file:// protocol that Pages understands.

What does this MR do?

Enable pages_serve_with_zip_file_protocol by default for self-managed instances.

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Tested manually on Omnibus nightly package gitlab-ee_13.9.0+rnightly.186879.66b31650-0_amd64.deb.

> Feature.enable(:pages_serve_from_migrated_zip)

Pages logs:

{"level":"info","msg":"GitLab internal pages status API connected successfully","time":"2021-02-19T00:58:49Z"}
{"host":"gitlab.jaimem.me:443","level":"trace","msg":"httptrace.ClientTrace.GetConn","time":"2021-02-19T01:07:48Z"}
{"connection_resumed":false,"error":null,"level":"trace","msg":"httptrace.ClientTrace.TLSHandshakeDone","time":"2021-02-19T01:07:48Z","version":772}
{"idle_time_ms":0,"level":"trace","msg":"httptrace.ClientTrace.GotConn","reused":false,"time":"2021-02-19T01:07:48Z","was_idle":false}
{"cache-control":"max-age=0, private, must-revalidate","client_name":"gitlab_internal_api","connection":"keep-alive","content-type":"application/json","date":"Fri, 19 Feb 2021 01:07:49 GMT","etag":"W/\"b0024b4718a918f251cd567782b0a41a\"","level":"trace","msg":"response","referrer-policy":"strict-origin-when-cross-origin","req_url":"https://gitlab.jaimem.me/api/v4/internal/pages?host=root.pages.jaimem.me","res_status_code":200,"server":"nginx","strict-transport-security":"max-age=31536000","time":"2021-02-19T01:07:49Z","vary":"Accept-Encoding;Origin","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","x-gitlab-feature-category":"pages","x-request-id":"01EYVX88HTVNW1RC5WNQJMGC0V","x-runtime":"0.290369"}
{"level":"debug","msg":"retrieval response sent","time":"2021-02-19T01:07:49Z"}
{"accept-ranges":"bytes","client_name":"zip_vfs","content-length":"1","content-range":"bytes 0-0/1080","content-type":"application/zip","last-modified":"Thu, 18 Feb 2021 07:54:15 GMT","level":"trace","msg":"response","req_url":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","res_status_code":206,"time":"2021-02-19T01:07:49Z"}
{"accept-ranges":"bytes","client_name":"zip_vfs","content-length":"1024","content-range":"bytes 56-1079/1080","content-type":"application/zip","last-modified":"Thu, 18 Feb 2021 07:54:15 GMT","level":"trace","msg":"response","req_url":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","res_status_code":206,"time":"2021-02-19T01:07:49Z"}
{"accept-ranges":"bytes","client_name":"zip_vfs","content-length":"299","content-range":"bytes 781-1079/1080","content-type":"application/zip","last-modified":"Thu, 18 Feb 2021 07:54:15 GMT","level":"trace","msg":"response","req_url":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","res_status_code":206,"time":"2021-02-19T01:07:49Z"}
{"error":null,"level":"trace","msg":"Root call","path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":null,"level":"trace","msg":"Lstat call","name":".","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":null,"level":"trace","msg":"Lstat call","name":"/index.html","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":null,"level":"trace","msg":"Lstat call","name":"/index.html","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":"file does not exist","level":"trace","msg":"Lstat call","name":"/index.html.br","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":"file does not exist","level":"trace","msg":"Lstat call","name":"/index.html.gz","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"accept-ranges":"bytes","client_name":"zip_vfs","content-length":"30","content-range":"bytes 70-99/1080","content-type":"application/zip","last-modified":"Thu, 18 Feb 2021 07:54:15 GMT","level":"trace","msg":"response","req_url":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","res_status_code":206,"time":"2021-02-19T01:07:49Z"}
{"error":null,"level":"trace","msg":"Open call","name":"/index.html","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":null,"level":"trace","msg":"Lstat call","name":"/index.html","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"accept-ranges":"bytes","client_name":"zip_vfs","content-length":"326","content-range":"bytes 150-475/1080","content-type":"application/zip","last-modified":"Thu, 18 Feb 2021 07:54:15 GMT","level":"trace","msg":"response","req_url":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","res_status_code":206,"time":"2021-02-19T01:07:49Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"","duration_ms":31,"host":"root.pages.jaimem.me","level":"info","method":"GET","msg":"access","pages_host":"root.pages.jaimem.me","pages_https":true,"pages_project_id":2,"pages_project_prefix":"/pages-html/","pages_project_serving_type":"zip","proto":"HTTP/1.0","referrer":"","remote_addr":"118.127.122.131","remote_ip":"118.127.122.131","status":200,"system":"http","time":"2021-02-19T01:07:49Z","ttfb_ms":31,"uri":"/pages-html/","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","written_bytes":632}
{"error":null,"level":"trace","msg":"Root call","path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":null,"level":"trace","msg":"Lstat call","name":"style.css","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":null,"level":"trace","msg":"Lstat call","name":"style.css","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":"file does not exist","level":"trace","msg":"Lstat call","name":"style.css.br","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":"file does not exist","level":"trace","msg":"Lstat call","name":"style.css.gz","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"accept-ranges":"bytes","client_name":"zip_vfs","content-length":"30","content-range":"bytes 492-521/1080","content-type":"application/zip","last-modified":"Thu, 18 Feb 2021 07:54:15 GMT","level":"trace","msg":"response","req_url":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","res_status_code":206,"time":"2021-02-19T01:07:49Z"}
{"error":null,"level":"trace","msg":"Open call","name":"style.css","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"error":null,"level":"trace","msg":"Lstat call","name":"style.css","root-path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-19T01:07:49Z","vfs":"zip"}
{"accept-ranges":"bytes","client_name":"zip_vfs","content-length":"194","content-range":"bytes 571-764/1080","content-type":"application/zip","last-modified":"Thu, 18 Feb 2021 07:54:15 GMT","level":"trace","msg":"response","req_url":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","res_status_code":206,"time":"2021-02-19T01:07:49Z"}
{"content_type":"text/css; charset=utf-8","correlation_id":"","duration_ms":0,"host":"root.pages.jaimem.me","level":"info","method":"GET","msg":"access","pages_host":"root.pages.jaimem.me","pages_https":true,"pages_project_id":2,"pages_project_prefix":"/pages-html/","pages_project_serving_type":"zip","proto":"HTTP/1.0","referrer":"https://root.pages.jaimem.me/pages-html/","remote_addr":"118.127.122.131","remote_ip":"118.127.122.131","status":200,"system":"http","time":"2021-02-19T01:07:49Z","ttfb_ms":0,"uri":"/pages-html/style.css","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","written_bytes":313}
{"content_type":"text/html; charset=utf-8","correlation_id":"","duration_ms":0,"error":"domain does not exist","host":"root.pages.jaimem.me","level":"info","method":"GET","msg":"access","pages_host":"root.pages.jaimem.me","pages_https":true,"proto":"HTTP/1.0","referrer":"https://root.pages.jaimem.me/pages-html/","remote_addr":"118.127.122.131","remote_ip":"118.127.122.131","status":404,"system":"http","time":"2021-02-19T01:07:49Z","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","written_bytes":3067}

Error unsupported protocol scheme

If GitLab Pages serves a 500 error page, and the logs show an error unsupported protocol scheme \"file\""

{"error":"Get \"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip\": unsupported protocol scheme \"file\"","level":"trace","msg":"Root call","path":"file:///var/opt/gitlab/gitlab-rails/shared/pages/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/pages_deployments/1/artifacts.zip","time":"2021-02-17T07:09:16Z","vfs":"zip"}

Users can disable this feature flag from the rails console and report an issue

> Feature.disable(:pages_serve_from_migrated_zip)

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #321677 (closed)

Edited by Kamil Trzciński

Merge request reports