Problems deploying nightly builds to dev

Recently we had to revert a change in gitlab-org/gitlab-ce!14148 because dev was breaking due to a missing theme_id column in the users table: https://sentry.gitlap.com/gitlab/devgitlaborg/issues/45326/

Upon inspection of the database, I did not see:

A column in the database
No entry in the schema_migrations for this migration

I noticed that when dev started failing 5 hours ago, the git version on the site was af930b8, which was a nightly build that did not contain the revert.

When I tried to run apt-get update, I saw this:

# sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
0% [2 InRelease gpgv 247 kB] [Waiting for headers] [1 InRelease 14.0 kB/102 kB 14%]Couldn't create tempfiles for splitting up /var/lib/apt/lists/azure.archive.ubuntu.com_ubuntu_dists_xenial_InErr:2 http://azure.archive.ubuntu.com/ubuntu xenial InRelease                      
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Get:3 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]     
Hit:4 https://download.ceph.com/debian-jewel xenial InRelease       
0% [4 InRelease gpgv 8,474 B] [3 InRelease 14.0 kB/102 kB 14%] [1 InRelease 42.5 kB/102 kB 42%]Couldn't create tempfiles for splitting up /var/lib/apt/lists/download.ceph.com_debian-jewel_distErr:4 https://download.ceph.com/debian-jewel xenial InRelease                                  
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Err:1 http://security.ubuntu.com/ubuntu xenial-security InReleasedn't create tempfiles for splitting up /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_InRelease
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Err:3 http://azure.archive.ubuntu.com/ubuntu xenial-updates InReleasepfiles for splitting up /var/lib/apt/lists/partial/azure.archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Get:5 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Err:5 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease/var/lib/apt/lists/partial/azure.archive.ubuntu.com_ubuntu_dists_xenial-backports_InRelease
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Hit:6 https://packages.gitlab.com/gitlab/nightly-builds/ubuntu xenial InRelease
Err:6 https://packages.gitlab.com/gitlab/nightly-builds/ubuntu xenial InReleaseb/apt/lists/packages.gitlab.com_gitlab_nightly-builds_ubuntu_dists_xenial_InRelease
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Fetched 306 kB in 0s (368 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://azure.archive.ubuntu.com/ubuntu xenial InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://download.ceph.com/debian-jewel xenial InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.ubuntu.com/ubuntu xenial-security InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com/gitlab/nightly-builds/ubuntu xenial InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease  Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease  Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease  Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch https://download.ceph.com/debian-jewel/dists/xenial/InRelease  Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch https://packages.gitlab.com/gitlab/nightly-builds/ubuntu/dists/xenial/InRelease  Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Some index files failed to download. They have been ignored, or old ones used instead.

This issue led me to discussion in https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1577926, which suggested the permissions in /tmp/ were wrong. Indeed, they were:

root@dev:/tmp# ls -ld /tmp
drwx------ 20 root root 12288 Sep  9 14:03 /tmp

Once I adjusted the permissions, apt-get update went through. I went ahead and ran apt-mark unhold gitlab-ce:

root@dev:/tmp# chmod 1777 /tmp
root@dev:/tmp# ls -ld /tmp
drwxrwxrwt 20 root root 12288 Sep  9 14:03 /tmp

For reference, I ran apt-get install gitlab-ce and saw this:

Unpacking gitlab-ce (9.5.4+rnightly.62514.4d70e77f-0) over (9.5.4+rnightly.62373.b4f84707-0) ...
Setting up gitlab-ce (9.5.4+rnightly.62514.4d70e77f-0) ...

It looks like the real reason dev failed was due to the config on dev:

RuntimeError
------------
Unsupported configuration detected in 'git_data_dirs' in /etc/gitlab/gitlab.rb.

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_shell.rb:37:in `block in parse_git_data_dirs'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_shell.rb:37:in `select'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_shell.rb:37:in `parse_git_data_dirs'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_shell.rb:22:in `parse_variables'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:168:in `block in generate_config'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:166:in `each'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:166:in `generate_config'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:26:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_shell.rb:

 30:      def parse_git_data_dirs
 31:        # Make sure we inform the user that they are using configuration that is
 32:        # not supported.
 33:        raise 'Encountered unsupported config key \'git_data_dir\' in /etc/gitlab/gitlab.rb.' if Gitlab['git_data_dir']
 34:  
 35:        git_data_dirs = Gitlab['git_data_dirs']
 36:        return unless git_data_dirs.any?
 37>>       git_data_dirs.select{ |k,v| raise 'Unsupported configuration detected in \'git_data_dirs\' in /etc/gitlab/gitlab.rb.' if v.is_a?(String)}
 38:  
 39:        gitaly_address = Gitaly.gitaly_address
 40:  
 41:        Gitlab['gitlab_shell']['git_data_directories'] ||= git_data_dirs
 42:        Gitlab['gitlab_rails']['repositories_storages'] ||=
 43:          Hash[Gitlab['gitlab_shell']['git_data_directories'].map do |name, data_directory|
 44:            shard_gitaly_address = data_directory['gitaly_address'] || gitaly_address
 45:  
 46:            defaults = { 'path' => File.join(data_directory['path'], 'repositories'), 'gitaly_address' => shard_gitaly_address }

Platform:
---------
x86_64-linux