Add Geo support for Packages::Debian::ProjectComponentFile
<!--
This template is based on a model named `PackagesDebianProjectComponentFile`.
To adapt this template, find and replace the following:
Template placeholders
- name: Cool Widgets
description: the human-readable name of the model (plural)
- name: Cool Widget
description: the human-readable name of the model (singular)
- name: cool_widgets
description: the snake-cased name of the model (plural)
- name: cool_widget
description: the snake-cased name of the model (singular)
- name: CoolWidget
description: the ActiveRecord class name of the model
- name: packagesDebianProjectComponentFile
description: the camel-cased name of the model
-->
## Replicate Debian ProjectComponentFiles- Blob
This issue is for implementing Geo replication and verification of Debian ProjectComponentFiles.
For more background, see [Geo self-service framework](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo/framework.md).
In order to implement and test this feature, you need to first [set up Geo locally](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/geo.md).
There are three main sections below. It is a good idea to structure your merge requests this way as well:
1. Modify database schemas to prepare to add Geo support for Debian ProjectComponentFiles
1. Implement Geo support of Debian ProjectComponentFiles behind a feature flag
1. Release Geo support of Debian ProjectComponentFiles
It is also a good idea to first open a proof-of-concept merge request. It can be helpful for working out kinks and getting initial support and feedback from the Geo team. As an example, see the [Proof of Concept to replicate Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56423).
You can look into the following examples of MRs for implementing replication/verification for a new blob type:
- [Add db changes](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60935) and [add verification for MR diffs using SSF](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309)
- [Verify Terraform state versions](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58800)
- [Verify LFS objects](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63981)
### Modify database schemas to prepare to add Geo support for Debian ProjectComponentFiles
#### Add the registry table to track replication and verification state
Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo.md#tracking-database) independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
- [x] Create the migration file in `ee/db/geo/migrate`:
```shell
bin/rails generate migration CreatePackagesDebianProjectComponentFileRegistry --database geo
```
- [x] Replace the contents of the migration file with the following. Note that we cannot add a foreign key constraint on `packages_debian_project_component_file_id` because the `packages_debian_project_component_files` table is in a different database. The application code must handle logic such as propagating deletions.
```ruby
# frozen_string_literal: true
class CreatePackagesDebianProjectComponentFileRegistry < Gitlab::Database::Migration[2.1]
def change
create_table :packages_debian_project_component_file_registry, id: :bigserial, force: :cascade do |t|
t.bigint :packages_debian_project_component_file_id, null: false
t.datetime_with_timezone :created_at, null: false
t.datetime_with_timezone :last_synced_at
t.datetime_with_timezone :retry_at
t.datetime_with_timezone :verified_at
t.datetime_with_timezone :verification_started_at
t.datetime_with_timezone :verification_retry_at
t.integer :state, default: 0, null: false, limit: 2
t.integer :verification_state, default: 0, null: false, limit: 2
t.integer :retry_count, default: 0, limit: 2, null: false
t.integer :verification_retry_count, default: 0, limit: 2, null: false
t.boolean :checksum_mismatch, default: false, null: false
t.binary :verification_checksum
t.binary :verification_checksum_mismatched
t.text :verification_failure, limit: 255
t.text :last_sync_failure, limit: 255
t.index :packages_debian_project_component_file_id, name: :index_packages_debian_project_component_file_registry_on_packages_debian_project_component_file_id, unique: true
t.index :retry_at
t.index :state
# To optimize performance of PackagesDebianProjectComponentFileRegistry.verification_failed_batch
t.index :verification_retry_at,
name: :packages_debian_project_component_file_registry_failed_verification,
order: "NULLS FIRST",
where: "((state = 2) AND (verification_state = 3))"
# To optimize performance of PackagesDebianProjectComponentFileRegistry.needs_verification_count
t.index :verification_state,
name: :packages_debian_project_component_file_registry_needs_verification,
where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))"
# To optimize performance of PackagesDebianProjectComponentFileRegistry.verification_pending_batch
t.index :verified_at,
name: :packages_debian_project_component_file_registry_pending_verification,
order: "NULLS FIRST",
where: "((state = 2) AND (verification_state = 0))"
end
end
end
```
- [x] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
- [x] Add the new table to the [database dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/database_dictionary.md) defined in [`ee/db/geo/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/db/geo/docs):
```yaml
table_name: packages_debian_project_component_file_registry
description: Description example
introduced_by_url: Merge request link
milestone: Milestone example
feature_categories:
- Feature category example
classes:
- Class example
gitlab_schema: gitlab_geo
```
- [x] Run Geo tracking database migrations:
```shell
bin/rake db:migrate:geo
```
- [x] Be sure to commit the relevant changes in `ee/db/geo/structure.sql` and the file created under `ee/db/geo/schema_migrations`
### Add verification state fields on the Geo primary site
The Geo primary site needs to checksum every replicable so secondaries can verify their own checksums. To do this, Geo requires fields on the Model. Add verification state fields to a separate table. Consult a database expert if needed.
#### Add verification state fields to a new table
- [x] Create the migration file in `db/migrate`:
```shell
bin/rails generate migration CreatePackagesDebianProjectComponentFileStates
```
- [x] Replace the contents of the migration file with:
```ruby
# frozen_string_literal: true
class CreatePackagesDebianProjectComponentFileStates < Gitlab::Database::Migration[2.1]
VERIFICATION_STATE_INDEX_NAME = "index_packages_debian_project_component_file_states_on_verification_state"
PENDING_VERIFICATION_INDEX_NAME = "index_packages_debian_project_component_file_states_pending_verification"
FAILED_VERIFICATION_INDEX_NAME = "index_packages_debian_project_component_file_states_failed_verification"
NEEDS_VERIFICATION_INDEX_NAME = "index_packages_debian_project_component_file_states_needs_verification"
def up
create_table :packages_debian_project_component_file_states do |t|
t.datetime_with_timezone :verification_started_at
t.datetime_with_timezone :verification_retry_at
t.datetime_with_timezone :verified_at
t.references :packages_debian_project_component_file,
null: false,
index: { unique: true },
foreign_key: { on_delete: :cascade }
t.integer :verification_state, default: 0, limit: 2, null: false
t.integer :verification_retry_count, default: 0, limit: 2, null: false
t.binary :verification_checksum, using: 'verification_checksum::bytea'
t.text :verification_failure, limit: 255
t.index :verification_state, name: VERIFICATION_STATE_INDEX_NAME
t.index :verified_at,
where: "(verification_state = 0)",
order: { verified_at: 'ASC NULLS FIRST' },
name: PENDING_VERIFICATION_INDEX_NAME
t.index :verification_retry_at,
where: "(verification_state = 3)",
order: { verification_retry_at: 'ASC NULLS FIRST' },
name: FAILED_VERIFICATION_INDEX_NAME
t.index :verification_state,
where: "(verification_state = 0 OR verification_state = 3)",
name: NEEDS_VERIFICATION_INDEX_NAME
end
end
def down
drop_table :packages_debian_project_component_file_states
end
end
```
- [x] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
- [x] If `packages_debian_project_component_files` is a high-traffic table, follow [the database documentation to use `with_lock_retries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/migration_style_guide.md#when-to-use-the-helper-method)
- [x] Add the new table to the [database dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/database_dictionary.md) defined in [`db/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/db/docs):
```yaml
---
table_name: packages_debian_project_component_file_states
description: Separate table for Debian ProjectComponentFile verification states
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/XXXXX
milestone: 'XX.Y'
feature_categories:
- geo_replication
classes:
- Geo::PackagesDebianProjectComponentFileState
gitlab_schema: gitlab_main
```
- [x] Run database migrations:
```shell
bin/rake db:migrate
```
- [x] Be sure to commit the relevant changes in `db/structure.sql` and the file under `db/schema_migrations`
That's all of the required database changes.
### Implement Geo support of Debian ProjectComponentFiles behind a feature flag
#### Step 1. Implement replication and verification
- [x] Add the following lines to the `packages_debian_project_component_file` model to accomplish some important tasks:
- Include `::Geo::ReplicableModel` in the `PackagesDebianProjectComponentFile` class, and specify the Replicator class `with_replicator Geo::PackagesDebianProjectComponentFileReplicator`.
- Include the `::Geo::VerifiableModel` concern.
- Delegate verification related methods to the `packages_debian_project_component_file_state` model.
- For verification, override some scopes to use the `packages_debian_project_component_file_states` table instead of the model table.
- Implement the `verification_state_object` method to return the object that holds
the verification details
- Override some methods to use the `packages_debian_project_component_file_states` table in verification-related queries.
At this point the `PackagesDebianProjectComponentFile` class should look like this:
```ruby
# frozen_string_literal: true
class PackagesDebianProjectComponentFile < ApplicationRecord
...
include ::Geo::ReplicableModel
include ::Geo::VerifiableModel
delegate(*::Geo::VerificationState::VERIFICATION_METHODS, to: :packages_debian_project_component_file_state)
with_replicator Geo::PackagesDebianProjectComponentFileReplicator
mount_uploader :file, PackagesDebianProjectComponentFileUploader
has_one :packages_debian_project_component_file_state, autosave: false, inverse_of: :packages_debian_project_component_file, class_name: 'Geo::PackagesDebianProjectComponentFileState'
# Override the `all` default if not all records can be replicated. For an
# example of an existing Model that needs to do this, see
# `EE::MergeRequestDiff`.
# scope :available_replicables, -> { all }
scope :available_verifiables, -> { joins(:packages_debian_project_component_file_state) }
scope :checksummed, -> {
joins(:packages_debian_project_component_file_state).where.not(packages_debian_project_component_file_states: { verification_checksum: nil })
}
scope :not_checksummed, -> {
joins(:packages_debian_project_component_file_state).where(packages_debian_project_component_file_states: { verification_checksum: nil })
}
scope :with_verification_state, ->(state) {
joins(:packages_debian_project_component_file_state)
.where(packages_debian_project_component_file_states: { verification_state: verification_state_value(state) })
}
def verification_state_object
packages_debian_project_component_file_state
end
...
class_methods do
extend ::Gitlab::Utils::Override
...
# @param primary_key_in [Range, PackagesDebianProjectComponentFile] arg to pass to primary_key_in scope
# @return [ActiveRecord::Relation<PackagesDebianProjectComponentFile>] everything that should be synced
# to this node, restricted by primary key
def replicables_for_current_secondary(primary_key_in)
# This issue template does not help you write this method.
#
# This method is called only on Geo secondary sites. It is called when
# we want to know which records to replicate. This is not easy to automate
# because for example:
#
# * The "selective sync" feature allows admins to choose which namespaces
# to replicate, per secondary site. Most Models are scoped to a
# namespace, but the nature of the relationship to a namespace varies
# between Models.
# * The "selective sync" feature allows admins to choose which shards to
# replicate, per secondary site. Repositories are associated with
# shards. Most blob types are not, but Project Uploads are.
# * Remote stored replicables are not replicated, by default. But the
# setting `sync_object_storage` enables replication of remote stored
# replicables.
#
# Search the codebase for examples, and consult a Geo expert if needed.
end
override :verification_state_model_key
def verification_state_model_key
:packages_debian_project_component_file_id
end
override :verification_state_table_class
def verification_state_table_class
PackagesDebianProjectComponentFileState
end
end
def packages_debian_project_component_file_state
super || build_packages_debian_project_component_file_state
end
...
end
```
- [x] Implement `PackagesDebianProjectComponentFile.replicables_for_current_secondary` above.
- [x] Ensure `PackagesDebianProjectComponentFile.replicables_for_current_secondary` is well-tested. Search the codebase for `replicables_for_current_secondary` to find examples of parameterized table specs. You may need to add more `FactoryBot` traits.
- [x] Add the following shared examples to `ee/spec/models/ee/packages/debian/project_component_file_spec.rb`:
```ruby
include_examples 'a verifiable model with a separate table for verification state' do
let(:verifiable_model_record) { build(:debian_project_component_file) } # add extra params if needed to make sure the record is in `Geo::ReplicableModel.verifiables` scope
let(:unverifiable_model_record) { build(:debian_project_component_file) } # add extra params if needed to make sure the record is NOT included in `Geo::ReplicableModel.verifiables` scope
end
```
- [x] Create `ee/app/replicators/geo/packages_debian_project_component_file_replicator.rb`. Implement the `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`, and implement the class method `.model` to return the `PackagesDebianProjectComponentFile` class:
- Implement the `replicable_title` and `replicable_title_plural` methods to
return the human-readable singular and pluralized title of the replicable,
which will be displayed in the UI and Rails console
```ruby
# frozen_string_literal: true
module Geo
class PackagesDebianProjectComponentFileReplicator < Gitlab::Geo::Replicator
include ::Geo::BlobReplicatorStrategy
extend ::Gitlab::Utils::Override
def self.model
::PackagesDebianProjectComponentFile
end
# @return [String] human-readable title.
def self.replicable_title
s_('Geo|Debian ProjectComponentFile')
end
# @return [String] pluralized human-readable title.
def self.replicable_title_plural
s_('Geo|Debian ProjectComponentFiles')
end
def carrierwave_uploader
model_record.file
end
override :verification_feature_flag_enabled?
def self.verification_feature_flag_enabled?
# We are adding verification at the same time as replication, so we
# don't need to toggle verification separately from replication. When
# the replication feature flag is off, then verification is also off
# (see `VerifiableReplicator.verification_enabled?`)
true
end
end
end
```
- [x] Generate the feature flag definition file by running the feature flag commands and following the command prompts:
```shell
bin/feature-flag --ee geo_packages_debian_project_component_file_replication --type development --group 'group::geo'
```
- [x] Add this replicator class to the method `replicator_classes` in
`ee/lib/gitlab/geo.rb`:
```ruby
REPLICATOR_CLASSES = [
::Geo::PackageFileReplicator,
::Geo::PackagesDebianProjectComponentFileReplicator
]
```
- [x] Create `ee/spec/replicators/geo/packages_debian_project_component_file_replicator_spec.rb` and perform the necessary setup to define the `model_record` variable for the shared examples:
```ruby
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::PackagesDebianProjectComponentFileReplicator, feature_category: :geo_replication do
let(:model_record) { build(:debian_project_component_file) }
include_examples 'a blob replicator'
end
```
- [x] Create `ee/app/models/geo/packages_debian_project_component_file_registry.rb`:
```ruby
# frozen_string_literal: true
module Geo
class PackagesDebianProjectComponentFileRegistry < Geo::BaseRegistry
include ::Geo::ReplicableRegistry
include ::Geo::VerifiableRegistry
MODEL_CLASS = ::PackagesDebianProjectComponentFile
MODEL_FOREIGN_KEY = :packages_debian_project_component_file_id
belongs_to :packages_debian_project_component_file, class_name: 'PackagesDebianProjectComponentFile'
end
end
```
- [x] Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`.
- [x] Add a custom factory name if needed in `def model_class_factory_name` in `ee/spec/support/helpers/ee/geo_helpers.rb`.
- [x] Update `it 'creates missing registries for each registry class'` in `ee/spec/workers/geo/secondary/registry_consistency_worker_spec.rb`.
- [x] Add `packages_debian_project_component_file_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`.
- [x] Create `ee/spec/factories/geo/packages_debian_project_component_file_registry.rb`:
```ruby
# frozen_string_literal: true
FactoryBot.define do
factory :geo_packages_debian_project_component_file_registry, class: 'Geo::PackagesDebianProjectComponentFileRegistry' do
packages_debian_project_component_file # This association should have data, like a file or repository
state { Geo::PackagesDebianProjectComponentFileRegistry.state_value(:pending) }
trait :synced do
state { Geo::PackagesDebianProjectComponentFileRegistry.state_value(:synced) }
last_synced_at { 5.days.ago }
end
trait :failed do
state { Geo::PackagesDebianProjectComponentFileRegistry.state_value(:failed) }
last_synced_at { 1.day.ago }
retry_count { 2 }
retry_at { 2.hours.from_now }
last_sync_failure { 'Random error' }
end
trait :started do
state { Geo::PackagesDebianProjectComponentFileRegistry.state_value(:started) }
last_synced_at { 1.day.ago }
retry_count { 0 }
end
trait :verification_succeeded do
synced
verification_checksum { 'e079a831cab27bcda7d81cd9b48296d0c3dd92ef' }
verification_state { Geo::PackagesDebianProjectComponentFileRegistry.verification_state_value(:verification_succeeded) }
verified_at { 5.days.ago }
end
trait :verification_failed do
synced
verification_failure { 'Could not calculate the checksum' }
verification_state { Geo::PackagesDebianProjectComponentFileRegistry.verification_state_value(:verification_failed) }
verification_retry_count { 1 }
verification_retry_at { 2.hours.from_now }
end
end
end
```
- [x] Create `ee/spec/models/geo/packages_debian_project_component_file_registry_spec.rb`:
```ruby
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::PackagesDebianProjectComponentFileRegistry, :geo, type: :model, feature_category: :geo_replication do
let_it_be(:registry) { create(:geo_packages_debian_project_component_file_registry) }
specify 'factory is valid' do
expect(registry).to be_valid
end
include_examples 'a Geo framework registry'
end
```
- [x] Add the following to `ee/spec/factories/packages/debian/component_file.rb`:
```ruby
# frozen_string_literal: true
FactoryBot.modify do
factory :debian_project_component_file do
trait :verification_succeeded do
with_file
verification_checksum { 'abc' }
verification_state { PackagesDebianProjectComponentFile.verification_state_value(:verification_succeeded) }
end
trait :verification_failed do
with_file
verification_failure { 'Could not calculate the checksum' }
verification_state { PackagesDebianProjectComponentFile.verification_state_value(:verification_failed) }
#
# Geo::VerifiableReplicator#after_verifiable_update tries to verify
# the replicable async and marks it as verification started when the
# model record is created/updated.
#
after(:create) do |instance, _|
instance.verification_failed!
end
end
end
end
```
If there is not an existing factory for the object in `spec/factories/packages/debian/component_file.rb`, wrap the traits in `FactoryBot.create` instead of `FactoryBot.modify`
- [x] Make sure the factory supports the `:remote_store` trait. If not, add something like
```ruby
trait :remote_store do
file_store { PackagesDebianProjectComponentFile::FileUploader::Store::REMOTE }
end
```
- [x] Make sure the factory also allows setting a `project` attribute. If the model does not have a direct relation to a project, you can use a `transient` attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
- [x] Following [the example of Merge Request Diffs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309) add a `Geo::PackagesDebianProjectComponentFileState` model in `ee/app/models/geo/packages_debian_project_component_file_state.rb`:
``` ruby
# frozen_string_literal: true
module Geo
class PackagesDebianProjectComponentFileState < ApplicationRecord
include ::Geo::VerificationStateDefinition
self.primary_key = :packages_debian_project_component_file_id
belongs_to :packages_debian_project_component_file, inverse_of: :packages_debian_project_component_file_state
validates :verification_state, :packages_debian_project_component_file, presence: true
end
end
```
- [x] Add a `factory` for `packages_debian_project_component_file_state`, in `ee/spec/factories/geo/packages_debian_project_component_file_states.rb`:
``` ruby
# frozen_string_literal: true
FactoryBot.define do
factory :geo_packages_debian_project_component_file_state, class: 'Geo::PackagesDebianProjectComponentFileState' do
packages_debian_project_component_file
trait :checksummed do
verification_checksum { 'abc' }
end
trait :checksum_failure do
verification_failure { 'Could not calculate the checksum' }
end
end
end
```
- [x] Add `[:packages_debian_project_component_file, :remote_store]` to `skipped` in `spec/support/shared_examples/lint_factories_shared_examples.rb`
#### Step 2. Implement metrics gathering
Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus` for display in the UI, and sent to Prometheus:
- [x] Add the following fields to Geo Node Status example responses in `doc/api/geo_nodes.md`:
- `packages_debian_project_component_files_count`
- `packages_debian_project_component_files_checksum_total_count`
- `packages_debian_project_component_files_checksummed_count`
- `packages_debian_project_component_files_checksum_failed_count`
- `packages_debian_project_component_files_synced_count`
- `packages_debian_project_component_files_failed_count`
- `packages_debian_project_component_files_registry_count`
- `packages_debian_project_component_files_verification_total_count`
- `packages_debian_project_component_files_verified_count`
- `packages_debian_project_component_files_verification_failed_count`
- `packages_debian_project_component_files_synced_in_percentage`
- `packages_debian_project_component_files_verified_in_percentage`
- [x] Add the same fields to `GET /geo_nodes/status` example response in
`ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json` and `ee/spec/fixtures/api/schemas/public_api/v4/geo_site_status.json`.
- [x] Add the following fields to the `Sidekiq metrics` table in `doc/administration/monitoring/prometheus/gitlab_metrics.md`:
```markdown
| `geo_packages_debian_project_component_files` | Gauge | XX.Y | Number of Debian ProjectComponentFiles on primary | `url` |
| `geo_packages_debian_project_component_files_checksum_total` | Gauge | XX.Y | Number of Debian ProjectComponentFiles to checksum on primary | `url` |
| `geo_packages_debian_project_component_files_checksummed` | Gauge | XX.Y | Number of Debian ProjectComponentFiles that successfully calculated the checksum on primary | `url` |
| `geo_packages_debian_project_component_files_checksum_failed` | Gauge | XX.Y | Number of Debian ProjectComponentFiles that failed to calculate the checksum on primary | `url` |
| `geo_packages_debian_project_component_files_synced` | Gauge | XX.Y | Number of syncable Debian ProjectComponentFiles synced on secondary | `url` |
| `geo_packages_debian_project_component_files_failed` | Gauge | XX.Y | Number of syncable Debian ProjectComponentFiles failed to sync on secondary | `url` |
| `geo_packages_debian_project_component_files_registry` | Gauge | XX.Y | Number of Debian ProjectComponentFiles in the registry | `url` |
| `geo_packages_debian_project_component_files_verification_total` | Gauge | XX.Y | Number of Debian ProjectComponentFiles to attempt to verify on secondary | `url` |
| `geo_packages_debian_project_component_files_verified` | Gauge | XX.Y | Number of Debian ProjectComponentFiles successfully verified on secondary | `url` |
| `geo_packages_debian_project_component_files_verification_failed` | Gauge | XX.Y | Number of Debian ProjectComponentFiles that failed verification on secondary | `url` |
```
- [x] Run the rake task `geo:dev:ssf_metrics` and commit the changes to `ee/config/metrics/object_schemas/geo_node_usage.json`
Debian ProjectComponentFile replication and verification metrics should now be available in the API, the `Admin > Geo > Sites` view, and Prometheus.
#### Step 3. Implement the GraphQL API
The GraphQL API is used by `Admin > Geo > Replication Details` views, and is directly queryable by administrators.
- [x] Add a new field to `GeoNodeType` in `ee/app/graphql/types/geo/geo_node_type.rb`:
```ruby
field :packages_debian_project_component_file_registries, ::Types::Geo::PackagesDebianProjectComponentFileRegistryType.connection_type,
null: true,
resolver: ::Resolvers::Geo::PackagesDebianProjectComponentFileRegistriesResolver,
description: 'Find Debian ProjectComponentFile registries on this Geo node. '\
'Ignored if `geo_packages_debian_project_component_file_replication` feature flag is disabled.',
experiment: { milestone: '15.5' } # Update the milestone
```
- [x] Add the new `packages_debian_project_component_file_registries` field name to the `expected_fields` array in `ee/spec/graphql/types/geo/geo_node_type_spec.rb`.
- [x] Create `ee/app/graphql/resolvers/geo/packages_debian_project_component_file_registries_resolver.rb`:
```ruby
# frozen_string_literal: true
module Resolvers
module Geo
class PackagesDebianProjectComponentFileRegistriesResolver < BaseResolver
type ::Types::Geo::GeoNodeType.connection_type, null: true
include RegistriesResolver
end
end
end
```
- [x] Create `ee/spec/graphql/resolvers/geo/packages_debian_project_component_file_registries_resolver_spec.rb`:
```ruby
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Resolvers::Geo::PackagesDebianProjectComponentFileRegistriesResolver, feature_category: :geo_replication do
it_behaves_like 'a Geo registries resolver', :geo_packages_debian_project_component_file_registry
end
```
- [x] Create `ee/app/finders/geo/packages_debian_project_component_file_registry_finder.rb`:
```ruby
# frozen_string_literal: true
module Geo
class PackagesDebianProjectComponentFileRegistryFinder
include FrameworkRegistryFinder
end
end
```
- [x] Create `ee/spec/finders/geo/packages_debian_project_component_file_registry_finder_spec.rb`:
```ruby
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::PackagesDebianProjectComponentFileRegistryFinder, feature_category: :geo_replication do
it_behaves_like 'a framework registry finder', :geo_packages_debian_project_component_file_registry
end
```
- [x] Create `ee/app/graphql/types/geo/packages_debian_project_component_file_registry_type.rb`:
```ruby
# frozen_string_literal: true
module Types
module Geo
# rubocop:disable Graphql/AuthorizeTypes -- because it is included
class PackagesDebianProjectComponentFileRegistryType < BaseObject
graphql_name 'PackagesDebianProjectComponentFileRegistry'
include ::Types::Geo::RegistryType
description 'Represents the Geo replication and verification state of a packages_debian_project_component_file'
field :packages_debian_project_component_file_id, GraphQL::Types::ID, null: false, description: 'ID of the Debian ProjectComponentFile.'
end
# rubocop:enable Graphql/AuthorizeTypes
end
end
```
- [x] Create `ee/spec/graphql/types/geo/packages_debian_project_component_file_registry_type_spec.rb`:
```ruby
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe GitlabSchema.types['PackagesDebianProjectComponentFileRegistry'], feature_category: :geo_replication do
it_behaves_like 'a Geo registry type'
it 'has the expected fields (other than those included in RegistryType)' do
expected_fields = %i[packages_debian_project_component_file_id]
expect(described_class).to have_graphql_fields(*expected_fields).at_least
end
end
```
- [x] Add integration tests for providing PackagesDebianProjectComponentFile registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`:
```ruby
it_behaves_like 'gets registries for', {
field_name: 'packagesDebianProjectComponentFileRegistries',
registry_class_name: 'PackagesDebianProjectComponentFileRegistry',
registry_factory: :geo_packages_debian_project_component_file_registry,
registry_foreign_key_field_name: 'packagesDebianProjectComponentFileId'
}
```
To allow the new replicable to resync and reverify via GraphQL:
- [x] Add the `PackagesDebianProjectComponentFileRegistryType` to the `GEO_REGISTRY_TYPE` constant in `ee/app/graphql/types/geo/registrable_type.rb`:
```ruby
GEO_REGISTRY_TYPES = {
::Geo::PackagesDebianProjectComponentFileRegistry => Types::Geo::PackagesDebianProjectComponentFileRegistryType
}
```
- [x] Include the `PackagesDebianProjectComponentFileRegistry` in the `let(:registry_classes)` variable of `ee/spec/graphql/types/geo/registry_class_enum_spec.rb`:
```ruby
let(:registry_classes) do
%w[
PACKAGES_DEBIAN_PROJECT_COMPONENT_FILE_REGISTRY
]
end
```
- [x] Include the new registry in the Rspec parameterized table of `ee/spec/support/shared_contexts/graphql/geo/registries_shared_context.rb`:
```ruby
# frozen_string_literal: true
RSpec.shared_context 'with geo registries shared context' do
using RSpec::Parameterized::TableSyntax
where(:registry_class, :registry_type, :registry_factory) do
Geo::PackagesDebianProjectComponentFileRegistry | Types::Geo::PackagesDebianProjectComponentFileRegistryType | :geo_packages_debian_project_component_file_registry
end
end
```
- [x] Update the GraphQL reference documentation:
```shell
bundle exec rake gitlab:graphql:compile_docs
```
Individual Debian ProjectComponentFile replication and verification data should now be available via the GraphQL API.
#### Step 4. Handle batch destroy
If batch destroy logic is implemented for a replicable, then that logic must be "replicated" by Geo secondaries. The easiest way to do this is use `Geo::BatchEventCreateWorker` to bulk insert a delete event for each replicable.
For example, if `FastDestroyAll` is used, then you may be able to [use `begin_fast_destroy` and `finalize_fast_destroy` hooks, like we did for uploads](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69763).
Or if a special service is used to batch delete records and their associated data, then you probably need to [hook into that service, like we did for job artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79530).
As illustrated by the above two examples, batch destroy logic cannot be handled automatically by Geo secondaries without restricting the way other teams perform batch destroys. It is up to you to produce `Geo::BatchEventCreateWorker` attributes before the records are deleted, and then enqueue `Geo::BatchEventCreateWorker` after the records are deleted.
- [x] Ensure that any batch destroy of this replicable is replicated to secondary sites
- [x] Regardless of implementation details, please verify in specs that when the parent object is removed, the new `Geo::Event` records are created:
```ruby
describe '#destroy' do
subject { create(:debian_project_component_file) }
context 'when running in a Geo primary node' do
let_it_be(:primary) { create(:geo_node, :primary) }
let_it_be(:secondary) { create(:geo_node) }
it 'logs an event to the Geo event log when bulk removal is used', :sidekiq_inline do
stub_current_geo_node(primary)
expect { subject.project.destroy! }.to change(Geo::Event.where(replicable_name: :packages_debian_project_component_file, event_name: :deleted), :count).by(1)
payload = Geo::Event.where(replicable_name: :packages_debian_project_component_file, event_name: :deleted).last.payload
expect(payload['model_record_id']).to eq(subject.id)
expect(payload['blob_path']).to eq(subject.relative_path)
expect(payload['uploader_class']).to eq('PackagesDebianProjectComponentFileUploader')
end
end
end
```
### Code Review
When requesting review from database reviewers:
- [x] Include a comment mentioning that the change is based on a documented template.
- [x] `replicables_for_current_secondary` and `available_replicables` may differ per Model. If their queries are new, then add [query plans](https://docs.gitlab.com/development/database_review/#query-plans) to the MR description. An easy place to gather SQL queries is your GDK's `log/test.log` when running tests of these methods.
### Release Geo support of Debian ProjectComponentFiles
- [ ] In the rollout issue you created when creating the feature flag, modify the Roll Out Steps:
- [ ] Cross out any steps related to testing on production GitLab.com, because Geo is not running on production GitLab.com at the moment.
- [ ] Add a step to `Test replication and verification of Debian ProjectComponentFiles on a non-GDK-deployment. For example, using GitLab Environment Toolkit`.
- [ ] Add a step to `Ping the Geo PM and EM to coordinate testing`. For example, you might add steps to generate Debian ProjectComponentFiles, and then a Geo engineer may take it from there.
- [ ] In `ee/config/feature_flags/development/geo_packages_debian_project_component_file_replication.yml`, set `default_enabled: true`
- [ ] In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `alpha` option for the released type:
```ruby
field :packages_debian_project_component_file_registries, ::Types::Geo::PackagesDebianProjectComponentFileRegistryType.connection_type,
null: true,
resolver: ::Resolvers::Geo::PackagesDebianProjectComponentFileRegistriesResolver,
description: 'Find Debian ProjectComponentFile registries on this Geo node. '\
'Ignored if `geo_packages_debian_project_component_file_replication` feature flag is disabled.',
experiment: { milestone: '15.5' } # Update the milestone
```
- [ ] Run `bundle exec rake gitlab:graphql:compile_docs` after the step above to regenerate the GraphQL docs.
- [ ] Add a row for Debian ProjectComponentFiles to the `Data types` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#data-types)
- [ ] Add a row for Debian ProjectComponentFiles to the `Limitations on replication/verification` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#limitations-on-replicationverification). If the row already exists, then update it to show that Replication and Verification is released in the current version.
issue