Mermaid broken since 14.7 upgrade

Summary

When opening an Epic with a mermaid graph (which worked well prior to the mentioned upgrade on 14.7) i see a error message where the mermaid graph should appear.

Steps to reproduce

Just open an epic with an embedded mermaid graph.

What is the current bug behavior?

An error message appears where the graph should be rendered.

What is the expected correct behavior?

The graph renders fine.

Relevant logs

The web-console on chrome outputs:

Refused to display '<URL>' in a frame because it set 'X-Frame-Options' to 'deny'.
chrome-error://chromewebdata/:1 Refused to display 'https://mygitlab.example.com/' in a frame because it set 'X-Frame-Options' to 'deny'.

Firefox shows a "Firefox Developer Edition Can’t Open This Page" message where the graph should be rendered and thats the output in the web-console:

GEThttps://mygitlab.example.com/-/sandbox/mermaid
[HTTP/2 200 OK 61ms]

This error page has no error code in its security info

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/-/sandbox/mermaid","format":"html","controller":"SandboxController","action":"mermaid","status":200,"time":"2022-01-24T18:40:53.959Z","params":[],"correlation_id":"01FT6P85K33W0EM9XRVMPZHFH9","meta.user":"tester","meta.caller_id":"SandboxController#mermaid","meta.remote_ip":"95.91.x","meta.feature_category":"not_owned","meta.client_id":"user/2","remote_ip":"95.91.x","user_id":2,"username":"tester","ua":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36","request_urgency":"default","target_duration_s":1,"redis_calls":9,"redis_duration_s":0.0016719999999999999,"redis_read_bytes":1205,"redis_write_bytes":1641,"redis_cache_calls":5,"redis_cache_duration_s":0.001207,"redis_cache_read_bytes":1014,"redis_cache_write_bytes":281,"redis_shared_state_calls":1,"redis_shared_state_duration_s":8.6e-05,"redis_shared_state_write_bytes":53,"redis_rate_limiting_calls":2,"redis_rate_limiting_duration_s":0.000199,"redis_rate_limiting_read_bytes":8,"redis_rate_limiting_write_bytes":132,"redis_sessions_calls":1,"redis_sessions_duration_s":0.00018,"redis_sessions_read_bytes":183,"redis_sessions_write_bytes":1175,"db_count":5,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":5,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"rack_attack_redis_count":1,"rack_attack_redis_duration_s":0.0003714399936143309,"cpu_s":0.075728,"mem_objects":83816,"mem_bytes":8394328,"mem_mallocs":31779,"mem_total_bytes":11746968,"pid":324886,"db_duration_s":0.02319,"view_duration_s":0.03362,"duration_s":0.08624}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/html; charset=utf-8","correlation_id":"01FT6P85K33W0EM9XRVMPZHFH9","duration_ms":101,"host":"mygitlab.example.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://mygitlab.example.com/groups/project_x/-/epics/13","remote_addr":"95.91.x:0","remote_ip":"95.91.x","route":"^/-/","status":200,"system":"http","time":"2022-01-24T19:40:53+01:00","ttfb_ms":101,"uri":"/-/sandbox/mermaid","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36","written_bytes":344}

==> /var/log/gitlab/nginx/gitlab_access.log <==
95.91.x - - [24/Jan/2022:19:40:53 +0100] "GET /-/sandbox/mermaid HTTP/2.0" 200 202 "https://mygitlab.example.com/groups/project_x/-/epics/13" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36" 1.70

Details of package version

Provide the package version installation details

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                        Version      Architecture Description
+++-===========================-============-============-============================================================
un  gitlab-ce                                 (no description available)
un  gitlab-ci-multi-runner                    (no description available)
un  gitlab-ci-multi-runner-beta               (no description available)
ii  gitlab-ee                   14.7.0-ee.0  amd64        GitLab Enterprise Edition (including NGINX, Postgres, Redis)
ii  gitlab-runner               14.7.0       amd64        GitLab Runner
un  gitlab-runner-beta                        (no description available)
un  gitlab-runner-fips                        (no description available)

Environment details

Operating System: Ubuntu 20.04.3 LTS
Installation Target, remove incorrect values:
- Bare Metal Machine
Installation Type, remove incorrect values:
- Upgrade from version 14.6.3
Is there any other software running on the machine: no
Is this a single or multiple node installation? single
Resources
- CPU: Intel(R) Xeon(R) CPU E3-1275 V2 @ 3.50GHz
- Memory total: 32g

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

external_url 'https://git.'
gitlab_rails['time_zone'] = 'Berlin'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'git@'
gitlab_rails['gitlab_email_display_name'] = ' Gitlab'
gitlab_rails['gitlab_email_reply_to'] = 'git@'
gitlab_rails['gitlab_default_can_create_group'] = false
gitlab_rails['gitlab_username_changing_enabled'] = false
gitlab_rails['gitlab_default_projects_features_wiki'] = false
gitlab_rails['gitlab_default_projects_features_snippets'] = false
gitlab_rails['gitlab_default_projects_features_builds'] = false
gitlab_rails['gitlab_default_projects_features_container_registry'] = false
gitlab_rails['incoming_email_enabled'] = true
gitlab_rails['incoming_email_address'] = "git+%{key}@"
gitlab_rails['incoming_email_email'] = "git@"
gitlab_rails['incoming_email_password'] = ""
gitlab_rails['incoming_email_host'] = "mail."
gitlab_rails['incoming_email_port'] = 143
gitlab_rails['incoming_email_start_tls'] = true
gitlab_rails['object_store']['enabled'] = false
gitlab_rails['object_store']['connection'] = {}
gitlab_rails['object_store']['storage_options'] = {}
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = nil
gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = nil
gitlab_rails['object_store']['objects']['lfs']['bucket'] = nil
gitlab_rails['object_store']['objects']['uploads']['bucket'] = nil
gitlab_rails['object_store']['objects']['packages']['bucket'] = nil
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = nil
gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = nil
gitlab_rails['lfs_enabled'] = true
gitlab_rails['backup_keep_time'] = 1209600
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail."
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "git@"
gitlab_rails['smtp_password'] = ""
gitlab_rails['smtp_domain'] = ""
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = true
registry_external_url 'https://registry.'
postgresql['shared_buffers'] = "1GB"
nginx['listen_addresses'] = ['*', '[::]']
pages_external_url "http://io."
gitlab_pages['enable'] = true
pages_nginx['listen_addresses'] = ['*', '[::]']
gitlab_kas['enable'] = true
mattermost_external_url 'https://chat.'
mattermost['enable'] = true
mattermost['sql_driver_name'] = 'postgres'
mattermost['gitlab_enable'] = true
mattermost['gitlab_id'] =     ""
mattermost['gitlab_secret'] = ""
mattermost['gitlab_scope'] = ""
mattermost['gitlab_auth_endpoint'] = "https://git./oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://git./oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://git./api/v4/user"
mattermost_nginx['listen_addresses'] = ['*', '[::]']
mattermost_nginx['custom_gitlab_mattermost_server_config'] = <<~FOCALBOARD
   location ~ /plugins/focalboard/ws/* {
       ## If you use HTTPS make sure you disable gzip compression
       ## to be safe against BREACH attack.
       gzip off;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_set_header X-Forwarded-Ssl on;
       proxy_pass http://gitlab_mattermost;
   }
FOCALBOARD
registry_nginx['listen_addresses'] = ['*', '[::]']
alertmanager['flags'] = {
  'cluster.advertise-address' => "127.0.0.1:9093"
}
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['support@'] # This should be an array of email addresses to add as contacts

Workarounds

1. Disable feature flag

Disabling the sandboxed_mermaid feature flag should resolve the issues and render the mermaid diagrams correctly.

sudo gitlab-rails console

Feature.disable(:sandboxed_mermaid)

Please take a look at https://docs.gitlab.com/ee/administration/feature_flags.html#how-to-enable-and-disable-features-behind-flags for detailed steps. The fix has been verified at gitlab#349892 (comment 819998588).

2. Enable Content Security Policy (for added security)

Enabling CSP for self-hosted instances with the example configuration should fix the issue and improves the overall security for the instance - https://docs.gitlab.com/omnibus/settings/configuration.html#content-security-policy

gitlab_rails['content_security_policy'] = {
    enabled: true,
    report_only: false,
    directives: {
      [...],
      frame_ancestors: "'self'",
    }
}

Edited Feb 08, 2022 by Dheeraj Joshi