Skip to content
Snippets Groups Projects

Document invalid metadata deletions being default enabled

Merged Sami Hiltunen requested to merge smh-default-enable-verifier into master
All threads resolved!
+ 15
4
@@ -268,14 +268,25 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
- This version removes `SanitizeConfidentialTodos` background migration which was [added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87908/diffs) in 15.6 and removed any user inaccessible Todos. Make sure that this migration is finished before upgrading to 15.9.
- As part of the [CI Partitioning effort](../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual. Make sure that this migration is finished before upgrading to 15.9.
- Praefect's metadata verifier's [invalid metadata deletion behavior](../administration/gitaly/praefect.md#enable-deletions) has been default enabled.
- Praefect's metadata verifier's [invalid metadata deletion behavior](../administration/gitaly/praefect.md#enable-deletions) is now enabled by default.
The metadata verifier goes through the replica records in Praefect's database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but in reality it doesn't exist on the disk. Once the metadata record is deleted, Praefect's reconciler will schedule a replication job to recreate the replica.
The metadata verifier processes replica records in the Praefect database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its
metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but, in reality, it doesn't exist on disk.
After the metadata record is deleted, Praefect's reconciler schedules a replication job to recreate the replica.
Due to past issues with the state management logic, there may be invalid metadata records in the database. These could exist for example due to incomplete deletions of repositories or partially completed renames. The verifier will delete these stale replica records of affected repositories. These repositories may show up as unavailable repositories in the metrics and `praefect dataloss` sub-command due to the replica records being removed. If you encounter such repositories, you may remove the repository using `praefect remove-repository` to get rid of repository's remaining records.
Because of past issues with the state management logic, there may be invalid metadata records in the database. These could exist, for example, because of incomplete
deletions of repositories or partially completed renames. The verifier deletes these stale replica records of affected repositories. These repositories may show up as
unavailable repositories in the metrics and `praefect dataloss` sub-command because of the replica records being removed. If you encounter such repositories, remove
the repository using `praefect remove-repository` to remove the repository's remaining records.
You can find repositories with invalid metadata records prior in GitLab 15.0 and later by searching for the log records outputted by the verifier. You can find an example log record [here](../administration/gitaly/praefect.md#repository-verification).
- Praefect's configuration structure in Omnibus GitLab [has been changed](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) to be consistent Praefect's own configuration structure used in source installs. Praefect's configuration is now under `praefect['configuration']` as a single hash. There are still other top-level keys in `praefect` used by Omnibus. Migrate by moving the configuration under the new structure. Below is the new structure with the old keys described in a comment above the key. Replace the `...` with the value from the old key. Default values are the same. If you haven't configured a value previously, you don't have to configure it now either. Remove the old keys from the configuration once migrated.
- Praefect configuration structure in Omnibus GitLab [has changed](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) to be consistent with the Praefect configuration structure
used in source installs. Praefect configuration is now under `praefect['configuration']` as a single hash. There are still other top-level keys in `praefect` used by
Omnibus GitLab.
Migrate by moving your existing configuration under the new structure. Below is the new structure with the old keys described in a comment above the key. Replace the
`...` with the value from the old key. Default values are the same. If you haven't configured a value previously, you don't need to configure it. Remove the old keys
from the configuration once migrated.
```ruby
praefect['configuration'] = {
Loading