Digital Ocean GitLab EE Marketplace offering is functionally broken

Summary

Digital Ocean offers GitLab EE as one of their DO Marketplace images: https://marketplace.digitalocean.com/apps/gitlab-enterprise-edition

The GitLab EE image they offer on their marketplace has customizations that appear to break all functionality and result in a frustrating and confusing user experience.

Proposed fix

The proposed fix is to simplify the image and installation process.

For the droplet image, include just an installation script that is kicked off on startup.
Installation script adds the GitLab package repository, and installs the then latest version of GitLab. (e.g. apt-get install gitlab-ee)
User then logs into the web UI and sets the root password.

The benefits of this solution is that there is little maintenance going forward, the package at install time is always up to date as it is installed on demand. The will mean that it will take a few minutes longer for the instance to start.

Based on adoption, we can make further improvements.

Steps to reproduce

Create droplet using DO Marketplace-provided GitLab EE
SSH to login to droplet and follow prompts, the script will either freeze with error about dpkg lock (Ctrl+C to exit) or enter a loop.
Visit login page. Regardless of whether you set a password for root in the DO first-run.sh setup script, one always gets prompted to set admin password in the Web UI on first visit.
Attempt to sign-in to GitLab (leading to 500 error)

What is the current bug behavior?

Spin up droplet:

Welcome to GitLab's One-Click GitLab Enterprise Edition Droplet.
To keep this Droplet secure, the UFW firewall is enabled. 
All ports are BLOCKED except 22 (SSH), 80 (HTTP), and 443 (HTTPS).

In a web browser, you can view:
 * The GitLab One-Click Quickstart guide: http://do.co/gitlab1804#start
 * The new GitLab site: http://167.170.110.0


GitLab is not configured. If you would like customize it yourself,
press 'n' now.

Okay to Configure GitLab (Y|n): Y
External URL (defaults to http://167.170.110.100): 
Enter the GitLab 'root' user password: secretpassword     
secretpassword: OK
Removing the landing page...
Running 'gitlab-ctl reconfigure', this will take a minute...
  wrote log to /var/log/gitlab_reconfigure.log
Setting the password...
  wrote log to /var/log/gitlab_set_pass.log
-----------------------------------------------------------------------------

You can access GitLab via:
    Web URL:  http://167.170.110.0
    User:     root
    Password: secretpassword

For more information about this 1-Click, see: http://do.co/gitlabapp

To enable LetsEncrypt TLS certificates, see: http://do.co/le-gitlab

Happy Coding!
-----------------------------------------------------------------------------

Followed again by:

GitLab is not configured. If you would like customize it yourself,
press 'n' now.

Okay to Configure GitLab (Y|n): n

Navigating to http://external_url, one is immediately prompted to set the root user password. After setting the password for root, attempts to log in as root with the password you've set will immediately lead to a 500 error.

gitlab-ctl reconfigure or uninstalling/reinstalling GitLab do not resolve the issue. 500 error on log in every time.

What is the expected correct behavior?

Admins and users are able to log in to GitLab and get past the /users/sign_in page.

Relevant logs

==> /var/log/gitlab/postgresql/current <==
2020-01-24_18:15:05.12659 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2020-01-24_18:15:05.12664 LOG:  database system was interrupted; last known up at 2019-11-21 06:02:06 GMT
2020-01-24_18:15:05.12665 LOG:  database system was not properly shut down; automatic recovery in progress
2020-01-24_18:15:05.12665 LOG:  invalid record length at 0/76CA4E8: wanted 24, got 0
2020-01-24_18:15:05.12665 LOG:  redo is not required
2020-01-24_18:15:05.12666 LOG:  database system is ready to accept connections
2020-01-24_18:15:05.67743 LOG:  received SIGHUP, reloading configuration files
2020-01-24_18:15:05.67746 LOG:  parameter "shared_buffers" cannot be changed without restarting the server
2020-01-24_18:15:05.67747 LOG:  parameter "effective_cache_size" changed to "3988MB"
2020-01-24_18:15:05.67747 LOG:  configuration file "/var/opt/gitlab/postgresql/data/postgresql.conf" contains errors; unaffected changes were applied

==> /var/log/gitlab/nginx/current <==
2020-01-24_18:15:07.38504 2020/01/24 16:34:53 [emerg] 20208#0: open() "/var/log/gitlab/nginx/gitlab_access.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38504 nginx: [alert] could not open error log file: open() "/var/opt/gitlab/nginx/logs/error.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38505 2020/01/24 16:34:54 [emerg] 20224#0: open() "/var/log/gitlab/nginx/gitlab_access.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38505 nginx: [alert] could not open error log file: open() "/var/opt/gitlab/nginx/logs/error.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38505 2020/01/24 16:34:55 [emerg] 20240#0: open() "/var/log/gitlab/nginx/gitlab_access.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38506 nginx: [alert] could not open error log file: open() "/var/opt/gitlab/nginx/logs/error.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38507 2020/01/24 16:34:56 [emerg] 20263#0: open() "/var/log/gitlab/nginx/gitlab_access.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38508 nginx: [alert] could not open error log file: open() "/var/opt/gitlab/nginx/logs/error.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38508 2020/01/24 16:34:57 [emerg] 20281#0: open() "/var/log/gitlab/nginx/gitlab_access.log" failed (2: No such file or directory)
2020-01-24_18:15:07.38508 nginx: [alert] could not open error log file: open() "/var/opt/gitlab/nginx/logs/error.log" failed (2: No such file or directory)

==> /var/log/gitlab/prometheus/current <==
2020-01-24_18:17:39.46786 level=info ts=2020-01-24T18:17:39.467Z caller=head.go:663 component=tsdb msg="WAL checkpoint complete" first=772 last=773 duration=433.499263ms
2020-01-24_18:18:05.41827 level=warn ts=2020-01-24T18:18:05.418Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:19:05.41427 level=warn ts=2020-01-24T18:19:05.414Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:20:05.47586 level=warn ts=2020-01-24T18:20:05.475Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:21:05.41607 level=warn ts=2020-01-24T18:21:05.415Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:22:05.41835 level=warn ts=2020-01-24T18:22:05.418Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:23:05.41528 level=warn ts=2020-01-24T18:23:05.415Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:24:05.41966 level=warn ts=2020-01-24T18:24:05.419Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:25:05.41786 level=warn ts=2020-01-24T18:25:05.417Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
2020-01-24_18:26:05.42235 level=warn ts=2020-01-24T18:26:05.422Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1


==> /var/log/gitlab/nginx/gitlab_access.log <==

210.160.90.0 - - [24/Jan/2020:18:16:53 +0000] "GET /users/sign_in HTTP/1.1" 200 4962 "http://160.170.110.0/users/password/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
210.160.90.0 - - [24/Jan/2020:18:16:53 +0000] "POST /users/password HTTP/1.1" 302 102 "http://160.170.110.0/users/password/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
210.160.90.0 - - [24/Jan/2020:18:16:59 +0000] "POST /users/sign_in HTTP/1.1" 500 2926 "http://160.170.110.0/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"

gitlab-workhorse/current

{
  "correlation_id": "vWs9LcETxs3",
  "duration_ms": 0,
  "host": "160.170.110.0",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "http://160.170.110.0/users/password/edit?reset_password_token=[FILTERED]",
  "remote_addr": "210.160.90.0:0",
  "remote_ip": "210.160.90.0",
  "status": 200,
  "system": "http",
  "time": "2020-01-24T18:16:35Z",
  "uri": "/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
  "written_bytes": 1611
}
{
  "correlation_id": "PVqj71yRlJ2",
  "duration_ms": 348,
  "host": "160.170.110.0",
  "level": "info",
  "method": "POST",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "http://160.170.110.0/users/password/edit?reset_password_token=[FILTERED]",
  "remote_addr": "210.160.90.0:0",
  "remote_ip": "210.160.90.0",
  "status": 302,
  "system": "http",
  "time": "2020-01-24T18:16:53Z",
  "uri": "/users/password",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
  "written_bytes": 102
}
{
  "correlation_id": "1F5zpNAL82a",
  "duration_ms": 422,
  "host": "160.170.110.0",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "http://160.170.110.0/users/password/edit?reset_password_token=[FILTERED]",
  "remote_addr": "210.160.90.0:0",
  "remote_ip": "210.160.90.0",
  "status": 200,
  "system": "http",
  "time": "2020-01-24T18:16:53Z",
  "uri": "/users/sign_in",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
  "written_bytes": 13068
}
{
  "correlation_id": "vfufpoYWsj2",
  "duration_ms": 1,
  "host": "160.170.110.0",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "http://160.170.110.0/users/sign_in",
  "remote_addr": "210.160.90.0:0",
  "remote_ip": "210.160.90.0",
  "status": 200,
  "system": "http",
  "time": "2020-01-24T18:16:53Z",
  "uri": "/assets/webpack/commons~pages.groups.omniauth_callbacks~pages.ldap.omniauth_callbacks~pages.omniauth_callbacks~pages~577d7818.32cfa2b3.chunk.js",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
  "written_bytes": 3631
}
{
  "correlation_id": "14ksYMg5Um4",
  "duration_ms": 316,
  "host": "160.170.110.0",
  "level": "info",
  "method": "POST",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "http://160.170.110.0/users/sign_in",
  "remote_addr": "210.160.90.0:0",
  "remote_ip": "210.160.90.0",
  "status": 500,
  "system": "http",
  "time": "2020-01-24T18:16:59Z",
  "uri": "/users/sign_in",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
  "written_bytes": 2926
}
{
  "correlation_id": "7qQG4Qk45T8",
  "duration_ms": 72,
  "host": "160.170.110.0",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "http://160.170.110.0/users/sign_in",
  "remote_addr": "210.160.90.0:0",
  "remote_ip": "210.160.90.0",
  "status": 301,
  "system": "http",
  "time": "2020-01-24T18:16:59Z",
  "uri": "/favicon.ico",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
  "written_bytes": 172
}

Details of package version

Customized package/image details

GitLab EE 12.4.3 (out of date, missing security fixes)
Custom script at first login `/opt/gitlab/first_login.sh`.
Three `/etc/gitlab/*.rb*` files used for configuration `gitlab.rb`, `gitlab.rb.digitalocean`, and `gitlab.rb.pre`. `gitlab.rb` template with defaults 
commented-out not included/available.

Environment details

Operating System: Ubuntu 18.04
Installation Target, remove incorrect values:
- VM: Digital Ocean
Installation Type, remove incorrect values:
- New Installation: Yes
Is there any other software running on the machine: DO-installed lighttpd
Is this a single or multiple node installation? Single
Resources
- CPU: 2-4
- Memory total: 4-8

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

# grep -v -e '^#' -e '^$' /etc/gitlab/gitlab.rb 
external_url "http://160.170.110.0
# grep -v -e '^#' -e '^$' /etc/gitlab/gitlab.rb.pre
nginx['listen_addresses'] = ["127.0.0.1"]
nginx['listen_port'] = 6080
# grep -v -e '^#' -e '^$' /etc/gitlab/gitlab.rb.digitalocean
#

Edited Apr 28, 2021 by silv