Certificates added to trusted-certs not getting handled if first reconfigure after adding them missed handling them

To reproduce

1. Install GitLab
2. Move /opt/gitlab/embedded/bin/c_rehash to /opt/gitlab/embedded/bin/c_rehash.bak
3. Add certificates to /etc/gitlab/trusted-certs.
4. Run reconfigure. See that the certificates didn't end up in /opt/gitlab/embedded/ssl/certs. (We don't warn users at all about c_rehash failing - tracked by #6569 (closed))
5. Move /opt/gitlab/embedded/bin/c_rehash.bak back to /opt/gitlab/embedded/bin/c_rehash
6. Run reconfigure. See that the certificates still didn't end up in /opt/gitlab/embedded/ssl/certs

Why

The block to move/symlink certificates gets run only if we detect new certificates have been added. However, our logic to detect this depends on the state of /etc/gitlab/trusted-certs and /opt/gitlab/embedded/ssl/certs at the beginning of reconfigure (during Chef's compile phase). With the following diff, we can see what happened

diff --git a/files/gitlab-cookbooks/package/libraries/certificate_helper.rb b/files/gitlab-cookbooks/package/libraries/certificate_helper.rb
index fe603f9d3..e6076f8ad 100644
--- a/files/gitlab-cookbooks/package/libraries/certificate_helper.rb
+++ b/files/gitlab-cookbooks/package/libraries/certificate_helper.rb
@@ -51,7 +51,10 @@ class CertificateHelper
   # If the number of files between the two directories is different
   # something got added so trigger the run
   def new_certificate_added?
-    return true unless File.exist?(@directory_hash_file)
+    unless File.exist?(@directory_hash_file)
+      puts "No directory hash file exists. Considering new certificates added"
+      return true
+    end
 
     stored_hash = File.read(@directory_hash_file)
     trusted_certs_dir_hash != stored_hash
@@ -60,7 +63,10 @@ class CertificateHelper
   def trusted_certs_dir_hash
     files = Dir[File.join(@trusted_certs_dir, "*"), File.join(@omnibus_certs_dir, "*")]
     files_modification_time = files.map { |name| File.stat(name).mtime if valid?(name) }
-    Digest::SHA1.hexdigest(files_modification_time.join)
+    hash = Digest::SHA1.hexdigest(files_modification_time.join)
+    puts "Computed hash value is #{hash}"
+
+    hash
   end
 
   # Get all files in /opt/gitlab/embedded/ssl/certs

1. First reconfigure, without c_rehash present

1. Status of certificate related directories after first reconfigure

1. Content of /var/opt/gitlab/trusted-certs-directory-hash after first reconfigure

1. Second reconfigure, after bringing back c_rehash

We can see that because the hash value remained same, the block was skipped. And unless the files inside /etc/gitlab/trusted-certs are changed (deleted and add backed on), this will be the case.

Possible fixes

1. Do not write to the hash file if symlinking/moving failed. So, this could be automatically fixed when we fix #6569 (closed) . Or,
2. Check if all the files in /etc/gitlab/trusted-certs has a symlink pointing towards to it from and check if there are any dangling symlinks in /opt/gitlab/embedded/ssl/certs. If either of this is true, proceed with the task in hand. If both are false, skip the task.