Adding secure files administration docs

5fa7bba4 · Darby Frey · Achilleas Pipinellis · 963bdc0e · 5fa7bba4 · 5fa7bba4
Commit 5fa7bba4 authored 1 year ago by Darby Frey Committed by Achilleas Pipinellis 1 year ago
--- a/doc/administration/object_storage.md
+++ b/doc/administration/object_storage.md
@@ -114,6 +114,7 @@ The following table lists the valid `objects` that can be used:
 | `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md)                    |
 | `terraform_state`  | [Terraform state files](terraform_state.md)                                |
 | `pages`            | [Pages](pages/index.md)                                             |
+| `ci_secure_files`  | [Project-level Secure Files](secure_files.md)                              |

 Within each object type, three parameters can be defined:

@@ -161,6 +162,7 @@ supported by consolidated form, refer to the following guides:

 | Object storage type | Supported by consolidated form? |
 |---------------------|------------------------------------------|
+| [Project-level Secure Files](secure_files.md#using-object-storage) | **{dotted-circle}** No |
 | [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No |
 | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No |
 | [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No |
@@ -463,7 +465,6 @@ The following example uses AWS S3 to enable object storage for all supported ser
   gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
   gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
   gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
-   gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
   gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
   ```

@@ -726,6 +727,7 @@ To migrate existing local data to object storage see the following guides:
 - [Dependency Proxy](packages/dependency_proxy.md#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage)
 - [Terraform state files](terraform_state.md#migrate-to-object-storage)
 - [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage)
+- [Project-level Secure Files](secure_files.md#migrate-to-object-storage)

 ## Transition to consolidated form


--- a/doc/administration/raketasks/check.md
+++ b/doc/administration/raketasks/check.md
@@ -121,12 +121,14 @@ Integrity checks are supported for the following types of file:

 - CI artifacts (introduced in GitLab 10.7.0)
 - LFS objects (introduced in GitLab 10.6.0)
+- Project-level Secure Files (introduced in GitLab 16.1.0)
 - User uploads (introduced in GitLab 10.6.0)

 **Omnibus Installation**

 ```shell
 sudo gitlab-rake gitlab:artifacts:check
+sudo gitlab-rake gitlab:ci_secure_files:check
 sudo gitlab-rake gitlab:lfs:check
 sudo gitlab-rake gitlab:uploads:check
 ```
@@ -135,6 +137,7 @@ sudo gitlab-rake gitlab:uploads:check

 ```shell
 sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production
+sudo -u git -H bundle exec rake gitlab:ci_secure_files:check RAILS_ENV=production
 sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production
 sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production
 ```
@@ -151,6 +154,7 @@ Variable  | Type    | Description

 ```shell
 sudo gitlab-rake gitlab:artifacts:check BATCH=100 ID_FROM=50 ID_TO=250
+sudo gitlab-rake gitlab:ci_secure_files:check BATCH=100 ID_FROM=50 ID_TO=250
 sudo gitlab-rake gitlab:lfs:check BATCH=100 ID_FROM=50 ID_TO=250
 sudo gitlab-rake gitlab:uploads:check BATCH=100 ID_FROM=50 ID_TO=250
 ```
@@ -415,3 +419,37 @@ To update these references to point to local storage:
   ```

 The script to [delete references to missing artifacts](check.md#delete-references-to-missing-artifacts) now functions correctly and cleans up the database.
+
+### Delete references to missing secure files
+
+`VERBOSE=1 gitlab-rake gitlab:ci_secure_files:check` detects when secure files:
+
+- Are deleted outside of GitLab.
+- Have references still in the GitLab database.
+
+When this scenario is detected, the Rake task displays an error message. For example:
+
+```shell
+Checking integrity of CI Secure Files
+- 1..15: Failures: 2
+  - Job SecureFile: 9: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/ci_secure_files/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/2022_06_30/8/9/distribution.cer>
+  - Job SecureFile: 15: Remote object does not exist
+Done!
+
+```
+
+To delete these references to missing local or remote secure files:
+
+1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session).
+1. Run the following Ruby code:
+
+   ```ruby
+   secure_files_deleted = 0
+   ::Ci::SecureFile.find_each do |secure_file|                    ### Iterate secure files
+     next if secure_file.file.file.exists?                        ### Skip if the file reference is valid
+     secure_files_deleted += 1
+     puts "#{secure_file.id}  #{secure_file.file.path} is missing."     ### Allow verification before destroy
+   #  secure_file.destroy!                                           ### Uncomment to actually destroy
+   end
+   puts "Count of identified/destroyed invalid references: #{secure_files_deleted}"
+   ```
--- a/doc/administration/secure_files.md
+++ b/doc/administration/secure_files.md
+---
+stage: Mobile
+group: Mobile DevOps
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Secure Files administration **(FREE)**
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8 [with a flag](feature_flags.md) named `ci_secure_files`. Disabled by default.
+> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/350748) in GitLab 15.7. Feature flag `ci_secure_files` removed.
+
+You can securely store up to 100 files for use in CI/CD pipelines as secure files.
+These files are stored securely outside of your project's repository and are not version controlled.
+It is safe to store sensitive information in these files. Secure files support both plain text
+and binary file types, and must be 5 MB or less.
+
+The storage location of these files can be configured using the options described below,
+but the default locations are:
+
+- `/var/opt/gitlab/gitlab-rails/shared/ci_secure_files` for installations using the Linux package.
+- `/home/git/gitlab/shared/ci_secure_files` for self-compiled installations.
+
+Use [external object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-terraform-state-dependency-proxy)
+configuration for [GitLab Helm chart](https://docs.gitlab.com/charts/) installations.
+
+## Disabling Secure Files
+
+You can disable Secure Files across the entire GitLab instance. You might want to disable
+Secure Files to reduce disk space, or to remove access to the feature.
+
+To disable Secure Files, follow the steps below according to your installation.
+
+Prerequisite:
+
+- You must be an administrator.
+
+**For Linux package installations**
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
+
+   ```ruby
+   gitlab_rails['ci_secure_files_enabled'] = false
+   ```
+
+1. Save the file and reconfigure GitLab:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+**For self-compiled installations**
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
+
+   ```yaml
+   ci_secure_files:
+     enabled: false
+   ```
+
+1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
+
+## Using local storage
+
+The default configuration uses local storage. To change the location where Secure Files
+are stored locally, follow the steps below.
+
+**For Linux package installations**
+
+1. To change the storage path for example to `/mnt/storage/ci_secure_files`, edit
+   `/etc/gitlab/gitlab.rb` and add the following line:
+
+   ```ruby
+   gitlab_rails['ci_secure_files_storage_path'] = "/mnt/storage/ci_secure_files"
+   ```
+
+1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation)
+1. Save the file and reconfigure GitLab:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+**For self-compiled installations**
+
+1. To change the storage path for example to `/mnt/storage/ci_secure_files`, edit
+   `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
+
+   ```yaml
+   ci_secure_files:
+     enabled: true
+     storage_path: /mnt/storage/ci_secure_files
+   ```
+
+1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source)
+   for the changes to take effect.
+
+## Using object storage **(FREE SELF)**
+
+Instead of storing Secure Files on disk, you should use [one of the supported object storage options](object_storage.md#supported-object-storage-providers).
+This configuration relies on valid credentials to be configured already.
+
+[Read more about using object storage with GitLab](object_storage.md).
+
+NOTE:
+This feature is not supported by consolidated object storage configuration.
+Adding support is proposed in [issue 414673](https://gitlab.com/gitlab-org/gitlab/-/issues/414673).
+
+### Object storage settings
+
+The following settings are:
+
+- Nested under `ci_secure_files:` and then `object_store:` on source installations.
+- Prefixed by `ci_secure_files_object_store_` on Linux package installations.
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `enabled` | Enable/disable object storage | `false` |
+| `remote_directory` | The bucket name where Secure Files are stored | |
+| `connection` | Various connection options described below | |
+
+### S3-compatible connection settings
+
+See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings).
+
+**For Linux package installations:**
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, but using
+   the values you want:
+
+   ```ruby
+   gitlab_rails['ci_secure_files_object_store_enabled'] = true
+   gitlab_rails['ci_secure_files_object_store_remote_directory'] = "terraform"
+   gitlab_rails['ci_secure_files_object_store_connection'] = {
+     'provider' => 'AWS',
+     'region' => 'eu-central-1',
+     'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
+     'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY'
+   }
+   ```
+
+   NOTE:
+   If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs:
+
+   ```ruby
+   gitlab_rails['ci_secure_files_object_store_connection'] = {
+     'provider' => 'AWS',
+     'region' => 'eu-central-1',
+     'use_iam_profile' => true
+   }
+   ```
+
+1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation)
+1. Save the file and reconfigure GitLab:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+1. [Migrate any existing local states to the object storage](#migrate-to-object-storage)
+
+**For self-compiled installations**
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
+
+   ```yaml
+   ci_secure_files:
+     enabled: true
+     object_store:
+       enabled: true
+       remote_directory: "ci_secure_files"  # The bucket name
+       connection:
+         provider: AWS  # Only AWS supported at the moment
+         aws_access_key_id: AWS_ACCESS_KEY_ID
+         aws_secret_access_key: AWS_SECRET_ACCESS_KEY
+         region: eu-central-1
+   ```
+
+1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
+1. [Migrate any existing local states to the object storage](#migrate-to-object-storage)
+
+### Migrate to object storage
+
+> [Introduced](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/readme/-/issues/125) in GitLab 16.1.
+
+WARNING:
+It's not possible to migrate Secure Files from object storage back to local storage,
+so proceed with caution.
+
+To migrate Secure Files to object storage, follow the instructions below.
+
+- For Linux package installations:
+
+  ```shell
+  sudo gitlab-rake gitlab:ci_secure_files:migrate
+  ```
+
+- For self-compiled installations:
+
+  ```shell
+  sudo -u git -H bundle exec rake gitlab:ci_secure_files:migrate RAILS_ENV=production
+  ```