Draft: Add troubleshooting step to update db_key_size

What does this MR do?

Adds a troubleshooting documentation about a problem that can result from !43950 (merged) (discussion) combined with a too short db_key_base.

This aggregates the support team's snippets "Bad Secrets check 2" & "Bad Secret Cleaner".

Related issues

• ZD 148744 (internal) (long-term solution & snippets)
• ZD 183620 (internal)

Author's checklist (required)

• Follow the Documentation Guidelines and Style Guide.
• If you have Developer permissions or higher:
  • Ensure that the product tier badge is added to doc's h1.
  • Apply the documentation label, plus:
    • The corresponding DevOps stage and group labels, if applicable.
    • development guidelines when changing docs under doc/development/*, CONTRIBUTING.md, or README.md.
    • development guidelines and Documentation guidelines when changing docs under development/documentation/*.
    • development guidelines and Description templates (.gitlab/*) when creating/updating issue and MR description templates.
  • Assign the designated Technical Writer.

Do not add the feature, frontend, backend, ~"bug", or database labels if you are only updating documentation. These labels will cause the MR to be added to code verification QA issues.

When applicable:

• Update the permissions table.
• Link docs to and from the higher-level index page, plus other related docs where helpful.
• Add the product tier badge accordingly.
• Add GitLab's version history note(s).
• Add/update the feature flag section.

Review checklist

All reviewers can help ensure accuracy, clarity, completeness, and adherence to the Documentation Guidelines and Style Guide.

1. Primary Reviewer

• Review by a code reviewer or other selected colleague to confirm accuracy, clarity, and completeness. This can be skipped for minor fixes without substantive content changes.

2. Technical Writer

• Technical writer review. If not requested for this MR, must be scheduled post-merge. To request for this MR, assign the writer listed for the applicable DevOps stage.
  • Ensure docs metadata are present and up-to-date.
  • Ensure Technical Writing and documentation are added.
  • Add the corresponding docs:: scoped label.
  • If working on UI text, add the corresponding UI Text scoped label.
  • Add twdoing when starting work on the MR.
  • Add twfinished if Technical Writing team work on the MR is complete but it remains open.

For more information about labels, see Technical Writing workflows - Labels.

For suggestions that you are confident don't need to be reviewed, change them locally and push a commit directly to save others from unneeded reviews. For example:

• Clear typos, like this is a typpo.
• Minor issues, like single quotes instead of double quotes, Oxford commas, and periods.

For more information, see our documentation on Merging a merge request.

3. Maintainer

1. Review by assigned maintainer, who can always request/require the above reviews. Maintainer's review can occur before or after a technical writer review.
2. Ensure a release milestone is set.
3. If there has not been a technical writer review, create an issue for one using the Doc Review template.

doc/ci/secrets/index.md

@@ -168,3 +168,94 @@ You can also specify some attributes for the resulting Vault tokens, such as tim
 IP address range, and number of uses. The full list of options is available in
 [Vault's documentation on creating roles](https://www.vaultproject.io/api/auth/jwt#create-role)
 for the JSON web token method.
+
+## Troubleshooting
+
+### Update `db_key_base` before upgrading to 13.6
+
+A GitLab instance with a `db_key_base` shorter than 32 characters
+will encounter a problem when upgrading to [GitLab 13.6](https://about.gitlab.com/releases/2020/11/22/gitlab-13-6-released/)
+because it [starts checking the `db_key_base` size](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43950/).
+
+```plaintext
+== 20201008013434 GenerateCiJwtSigningKey: migrating ==========================
+STDERR:
+---- End output of "bash"  "/tmp/chef-script..." ----
+Ran "bash"  "/tmp/chef-script..." returned 1
+
+Deprecations:
+...
+
+===
+There was an error running gitlab-ctl reconfigure. Please check the output above for more
+details.
+===
+
+dpkg: error processing package gitlab-ee (--configure):
+ installed gitlab-ee package post-installation script subprocess returned error exit status 1
+E: Sub-process /usr/bin/dpkg returned an error code (1)
+```
+
+or
+
+```plaintext
+There was an error running gitlab-ctl reconfigure:
+
+Caused by:
+ArgumentError: key must be 32 bytes or longer
+...
+bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
+---- Begin output of "bash"  "/tmp/chef-script..." ----
+STDOUT: rake aborted!
+StandardError: An error has occurred, this and all later migrations canceled:
+...
+```
+
+#### Quick work-around
+
+This problem can be addressed by skipping the related migration.
+Executing the following in a
+[Rails console](../../administration/operations/rails_console.md):
+
+```ruby
+ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations VALUES ('20201008013434')")
+```
+
+After running `sudo gitlab-ctl reconfigure` to complete the upgrade,
+the above can be reverted with
+
+```ruby
+ActiveRecord::Base.connection.execute("DELETE FROM schema_migrations WHERE version='20201008013434')")
+```
+
+This workaround has the known side-effect of
+[the predefined `CI_JOB_JWT` variable](../variables/predefined_variables.md)
+not being available, which is used mostly for integration with Hashicorp Vault.
+
+#### Long-term solution
+
+The long-term solution is to update the `db_key_base` and
+to force a reset of each depent secret, key and token.
+You can do so as follows:
+
+1. Backup [your `gitlab-secrets.json` file](../../raketasks/backup_restore.md#storing-configuration-files).
+1. Using a text editor, fill the short `db_key_base` value in `gitlab-secrets.json` with `0`s to the desired length (64, 128, etc.)
+1. Run `sudo gitlab-ctl reconfigure` and `sudo gitlab-ctl restart`
+1. Check the output of `sudo gitlab-rails runner "puts Settings.attr_encrypted_db_key_base.size"`.
+  If that shows the same length you extended `db_key_base` to earlier,
+  it's safe to proceed.
+1. Run the following script to check the affected secrets:
+
+    ```shell
+    curl --output /tmp/secrets-check.rb "https://gitlab.com/snippets/1800209/raw"
+    sudo gitlab-rails runner /tmp/secrets-check.rb
+    ```
+
+1. Proceed with resetting them by running with the following script:
+
+    ```shell
+    curl --output /tmp/secrets-reset.rb "https://gitlab.com/snippets/1876342/raw"
+    sudo gitlab-rails runner /tmp/secrets-reset.rb
+    ```
+
+1. Run `sudo gitlab-ctl reconfigure` and `sudo gitlab-ctl restart`

[Katrin Leinweber]

@wchandler & @aciciu: Would inserting the script code here & below help to figure out the correct steps?

[Nick Gaskill]

@katrinleinweber Is this good to merge, or does it need further review?

[Katrin Leinweber]

@ngaskill: Yes to the latter. This needs more input from support engineers. @mrjkc is planning a call with them.