Draft: Add troubleshooting step to update db_key_size

Hi @wchandler, @aciciu, @atanayno & @krasio You all pioneered some aspect of this MR :-) Thank you!

For 183620 (internal) I think compiling your earlier advice into a single docu section would be beneficial for other customers.

I'm testing this just now, but could you please comment as well? In particular: Does anything additionally need to be done after running the 2 scripts/snippets? Are any kind of data or settings lost during such a key rotation?

I've tested the fact that a too short key blocks the upgrade to 13.6 and the fact that the instance can then be rescued by these steps. Afterwards, I noticed that the Runner token was different, so registering Runners is probably necessary, isn't it?

Hi @aqualls! I'm assigning this already, because I'll be OoO until early January. There is no hurry WRT your review, until the content has been confirmed by support or ~"group::configure".

@katrinleinweber Thanks! The workaround section (skipping migration 20201008013434) looks good. Not sure about the db_key_base rotation, I am surprised this will work - once we change the key how are we able to decrypt secrets in order to encrypt them again? Are those trailing 0s ignored (but if this is the case then why should we decrypt/encrypt at all)?

Also the list of secrets in table_column_combos is just some of the secrets encrypted using db_key_base, there are others and this script will leave them untouched.

@wchandler would be best to ask about this ;-) The "Bad Secret Cleaner" script mentioned above does the de- and encryption in one go, as far as I understood it.

@katrinleinweber in general we have to delete the secret. My vague recollection is that items that were just added may still have the original value available to re-encrypt, which is what the script checks for, but I could be wrong.

@wchandler: OK, thanks! I understand that the You can do so as follows I drafted would therefore not help for older encrypted items. Should we instead replace all those steps with advice to only increase the key size, check which items would be affected and re-create those?

Or, a way more drastic solution: Set up a fresh instance and import a backup from the one with a too short key?

@krasio customer would like to have the CI_JOB_JWT variable for the Hashicorp Vault therefore skipping migration is not a suitable solution for them, would there be any other solution in order to solve this problem?

@mrjkc Unfortunately I can't think of anything that will help them at the moment.

We may consider changing the migration and code to use attr_encrypted_db_key_base_32 instead of attr_encrypted_db_key_base_truncated for this attribute. This should work for both instances that have shorter db_key_base (< 32) and instances that have proper ones (and had the secret generated already). If you want to create an issue keep in mind Secrets Management was recently moved under ~"group::configure".

BTW there are many other encrypted attributes that rely on the truncated key, I imagine there will be a lot of broken functionality for such instances?

BTW there are many other encrypted attributes that rely on the truncated key, I imagine there will be a lot of broken functionality for such instances?

yup, we would have to re add the data that can not be decrypted with the new key. I am working at the moment to see if we can rotate the key somehow and re-encrypt existing data with the new key, without having to reset the data and add it again.

added customer label

mentioned in merge request !43950 (merged)

added 9 commits

0f965b06...19d08365 - 6 commits from branch master
1379bb39 - Add troubleshooting step to update db_key_size
edb4ec0b - Use snippet URLs, instead of listing their code
b98c5164 - Show both upgrade and reconfigure error

Compare with previous version

changed the description

added Category:Secrets Management groupconfigure [DEPRECATED] labels

added devopsconfigure [DEPRECATED] label

added Support Team Contributions label

assigned to @aqualls

1 Warning

This merge request includes more than 10 commits. Each commit should meet the following criteria:

Have a well-written commit message.
Has all tests passing when used on its own (e.g. when using git checkout SHA).
Can be reverted on its own without also requiring the revert of commit that came before it.
Is small enough that it can be reviewed in isolation in under 30 minutes or so.

If this merge request contains commits that do not meet this criteria and/or contains intermediate work, please rebase these commits into a smaller number of commits or split this merge request into multiple smaller merge requests.

	1 Message
	This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

Documentation review

The following files require a review from a technical writer:

doc/ci/secrets/index.md
doc/update/README.md

The review does not need to block merging this merge request. See the:

Technical Writers assignments for the appropriate technical writer for this review.
Documentation workflows for information on when to assign a merge request for review.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger