Document how to restore Terraform state files from backups

Goal

Document how to restore Terraform state files from backups assuming the encrypted state files and the GitLab DB are available.

Proposal

There was a discussion in Slack where @tigerwnz summarized the process

If they have still have the database records for the files somewhere, then I think the terraform_state_versions table and terraform_states tables would be a good start to trace an S3 path back to a project. Otherwise, it looks like the first three parts of the hash are derived from the hexadecimal SHA2 digest of the project ID. So if they no longer have the access to the database records, then their best bet is to hash all of their project IDs, creating a reverse index sha2(project_id) -> project_id and identifying the state like that.

they will need access to the database records. Along with the hash of the project ID described above, each state has a unique ID (uuid column on the terraform_states table), this is also used to define the path. It might be easier with an example. Imagine a state belonging to project ID 12345. It has a UUID of example-uuid, and the sha2 hash of 12345 is 5994471abb01112afcc18159f6cc74b4f511b99806da59b3caf5a9c173cacfc5. The folder structure would be as follows:

terraform/                                                                  <- configured terraform storage directory
├─ 59/                                                                      <- first and second character of project ID hash
|  ├─ 94/                                                                   <- third and fourth character of project ID hash
|  |  ├─ 5994471abb01112afcc18159f6cc74b4f511b99806da59b3caf5a9c173cacfc5/  <- full project ID hash
|  |  |  ├─ example-uuid/                                                   <- state UUID
|  |  |  |  ├─ 1.tf                                                         <- individual state versions
|  |  |  |  ├─ 2.tf
|  |  |  |  ├─ 3.tf

tiger 7 hours ago As for decryption, the state files are encrypted with a key derived from the db_key_base application setting (https://docs.gitlab.com/ee/development/application_secrets.html#secret-entries) and the project ID. If the db_key_base is not available, decryption will not be possible. The encryption key is defined here: https://gitlab.com/gitlab-org/gitlab/-/blob/e0137111fbbd28316f38da30075aba641e702b98/app/uploaders/terraform/state_uploader.rb#L43 Files are encrypted using Lockbox (https://github.com/ankane/lockbox), their documentation should be able to help with decrypting files manually.