## Summary This issue is to roll out and measure the impact of enabling Geo features (in 3 stages, initially only on staging): * Phase 1 rollout - enable metrics worker and checksumming on a small table & measure impact * Phase 2 rollout - enable checksumming on medium sized tables & measure impact * Phase 3 rollout - enable checksumming on largest remaining tables & measure impact Originally discussed during the Org Mover sync on 24th Oct 2024 - (see [point 3 in the notes here](https://docs.google.com/document/d/1bScsA2LmIfiLJtDdq79EdwTSGFdQxlUYxHjOtotU3pY/edit?tab=t.0)) After that, we can begin rolling out on production. ## Owners - Most appropriate Slack channel to reach out to: `#f_org_mover` / `#geo_lounge` - Best individuals to reach out to: `@s_murray` / `@mkozono` / `@dbalexandre` ## Metrics to measure Key metrics we want to measure and where to monitor these:- * Verification State Table Sizes - https://dashboards.gitlab.net/d/fdzkd1y2qpg5cf/checksumming?orgId=1 * Any other errors/increase visible in https://dashboards.gitlab.net - specifically: * cpu saturation, disk IO saturation, saturation of the sidekiq worker pool (or potentially the pgbouncer pool backing it) if too many jobs start, and long running vacuums ## Staging Table Sizes Staging table sizes are recorded below. We also have this dashboard set up to display the verification state table sizes - https://dashboards.gitlab.net/d/fdzkd1y2qpg5cf/checksumming?orgId=1 | Replicable | Source Table | Source Table Size | Verification State Table | Verification Table Size (Before) | Verification Table Size (After) | Source Table Row count | - | |------------|-------|------------|--------------------------|----------------------------------|----------------------------------|----------------------------------|--------------------------------| | MergeRequestDiff | merge_request_diffs | 2260 MB | merge_request_diff_details | 752 kB | | 9759196 | X | | ProjectRepository | projects | 1423 MB | project_states | 704 kB | | 3341788 | X | | LfsObject | lfs_objects | 555 MB | lfs_object_states | 358 MB | | 2843629 | X | | Upload | uploads | 295 MB | upload_states | 90 MB | | 1441495 | X | | DesignManagement::Repository | design_management_repositories | 223 MB | design_management_repository_states | 8192 bytes | | 3335232 | X | | Projects::WikiRepository | project_wiki_repositories | 202 MB | wiki_repository_states | 1072 kB | | 3341642 | X | | SnippetRepository | snippet_repositories | 18 MB | snippet_repositories | 18 MB | | 49051 | X | | Packages::PackageFile | packages_package_files | 5528 kB | packages_package_files | 5528 kB | | 1093 | X | | ContainerRepository | container_repositories | 5712 kB | container_repository_states | 8192 bytes | | 28434 | X | | PagesDeployment | pages_deployments | 4736 kB | pages_deployment_states | N/A | | 28593 | X | | GroupWikiRepository | group_wiki_repositories | 352 kB | group_wiki_repository_states | 16 kB | | 294 | X | | DependencyProxy::Blob | dependency_proxy_blobs | 272 kB | dependency_proxy_blob_states | N/A | | 1074 | X | | DependencyProxy::Manifest | dependency_proxy_manifests | 40 kB | dependency_proxy_manifest_states | N/A | | 1074 | X | | Terraform::StateVersion | terraform_state_versions | 8192 bytes | terraform_state_versions | 8192 bytes | | 22 | X | | Ci::JobArtifact | ci_job_artifacts | N/A | ci_job_artifact_states | 358 MB | | 2257449 | X | | Ci::PipelineArtifact | ci_pipeline_artifacts | N/A | ci_pipeline_artifacts | N/A | | 37 | X | | Ci::SecureFile | ci_secure_files | N/A | ci_secure_file_states | N/A | | 842 | X | ## Rollout Steps (staging) Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Phase 1 - Enable metrics worker and checksumming on a small table - [x] Enable the Geo::MetricsUpdateWorker feature with `/chatops run feature set geo_metrics_update_worker true --staging` - **Note:** Enabled at 1200 UTC - 12th Nov 2024 - [x] Monitor that the error rates did not increase - [x] Enable the force primary checksumming feature flag on small tables - [x] `/chatops run feature set geo_ci_secure_file_force_primary_checksumming true --staging` - **Note:** Enabled at 1400 UTC - 12th Nov 2024 - [ ] `/chatops run feature set geo_packages_nuget_symbol_force_primary_checksumming true --staging` - [x] Monitor that the error rates did not increase ### Phase 2 - Enable checksumming on remaining medium sized tables - [x] Enable the force primary checksumming feature flag on medium sized tables - [x] `/chatops run feature set geo_dependency_proxy_blob_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_dependency_proxy_manifest_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_group_wiki_repository_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_pages_deployment_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_pipeline_artifact_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_terraform_state_version_force_primary_checksumming true --staging` - [x] Monitor that error rates did not increase - [x] Monitor impact on metrics defined in the monitoring section for a few hours ### Phase 3 - Enable checksumming on the largest tables - [x] Enable the force primary checksumming feature flag on largest tables - [x] `/chatops run feature set geo_package_file_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_snippet_repository_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_container_repository_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_project_wiki_repository_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_design_management_repository_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_upload_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_lfs_object_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_project_repository_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_merge_request_diff_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_job_artifact_force_primary_checksumming true --staging` - [x] `/chatops run feature set geo_packages_helm_metadata_cache_force_primary_checksumming true --staging` - [x] Monitor that error rates did not increase - [x] Monitor impact on metrics defined in the monitoring section for a few hours ## Rollback Steps - [ ] Disable the feature flags on staging: ``` ### Phase 1 /chatops run feature set geo_metrics_update_worker false --staging /chatops run feature set geo_ci_secure_file_force_primary_checksumming false --staging ### Phase 2 /chatops run feature set geo_dependency_proxy_blob_force_primary_checksumming false --staging /chatops run feature set geo_dependency_proxy_manifest_force_primary_checksumming false --staging /chatops run feature set geo_group_wiki_repository_force_primary_checksumming false --staging /chatops run feature set geo_pages_deployment_force_primary_checksumming false --staging /chatops run feature set geo_pipeline_artifact_force_primary_checksumming false --staging /chatops run feature set geo_terraform_state_version_force_primary_checksumming false --staging ### Phase 3 /chatops run feature set geo_snippet_repository_force_primary_checksumming false --staging /chatops run feature set geo_project_wiki_repository_force_primary_checksumming false --staging /chatops run feature set geo_design_management_repository_force_primary_checksumming false --staging /chatops run feature set geo_container_repository_force_primary_checksumming false --staging /chatops run feature set geo_package_file_force_primary_checksumming false --staging /chatops run feature set geo_upload_force_primary_checksumming false --staging /chatops run feature set geo_lfs_object_force_primary_checksumming false --staging /chatops run feature set geo_project_repository_force_primary_checksumming false --staging /chatops run feature set geo_merge_request_diff_force_primary_checksumming false --staging /chatops run feature set geo_job_artifact_force_primary_checksumming false --staging /chatops run feature set geo_packages_helm_metadata_cache_force_primary_checksumming false --staging ``` #### Data rollback (optional) Optional step to remove any data created during the process. ```ruby # Remove data created by the metrics worker GeoNodeStatus.connection.truncate(GeoNodeStatus.table_name) # Remove verification state data on separate _state tables Geo::ProjectState.connection.truncate(Geo::ProjectState.table-name) Geo::ProjectRepositoryState.connection.truncate(Geo::ProjectRepositoryState.table_name) Geo::LfsObjectState.connection.truncate(Geo::LfsObjectState.table_name) Geo::UploadState.connection.truncate(Geo::UploadState.table_name) Geo::DesignManagementRepositoryState.connection.truncate(Geo::DesignManagementRepositoryState.table_name) Geo::WikiRepositoryState.connection.truncate(Geo::WikiRepositoryState.table_name) Geo::ContainerRepositoryState.connection.truncate(Geo::ContainerRepositoryState.table_name) Geo::PagesDeploymentState.connection.truncate(Geo::PagesDeploymentState.table_name) Geo::GroupWikiRepositoryState.connection.truncate(Geo::GroupWikiRepositoryState.table_name) Geo::DependencyProxyBlobState.connection.truncate(Geo::DependencyProxyBlobState.table_name) Geo::DependencyProxyManifestState.connection.truncate(Geo::DependencyProxyManifestState.table_name) Geo::JobArtifactState.connection.truncate(Geo::JobArtifactState.table_name) Geo::CiSecureFileState.connection.truncate(Ci::SecureFileState.table_name) Geo::PackagesNugetSymbolState.connection.truncate(Geo::PackagesNugetSymbolState.table_name) Geo::PackagesHelmMetadataCacheState.connection.truncate(Geo::PackagesHelmMetadataCacheState.table_name) # Update/remove verification state data added to existing source tables # (Probably not worth it, and can time out on large tables) MergeRequestDiffDetail.update_all( verification_retry_at: nil, verified_at: nil, verification_retry_count: nil, verification_checksum: nil, verification_failure: nil, verification_state: 0, verification_started_at: nil ) SnippetRepository.update_all( verification_retry_at: nil, verified_at: nil, verification_retry_count: nil, verification_checksum: nil, verification_failure: nil, verification_state: 0, verification_started_at: nil ) Packages::PackageFile.update_all( verification_retry_at: nil, verified_at: nil, verification_retry_count: nil, verification_checksum: nil, verification_failure: nil, verification_state: 0, verification_started_at: nil ) Terraform::StateVersion.update_all( verification_retry_at: nil, verified_at: nil, verification_retry_count: nil, verification_checksum: nil, verification_failure: nil, verification_state: 0, verification_started_at: nil ) Ci::PipelineArtifact.update_all( verification_retry_at: nil, verified_at: nil, verification_retry_count: nil, verification_checksum: nil, verification_failure: nil, verification_state: 0, verification_started_at: nil ) ``` ### Rollout Summary #### Phase 1 summary (complete) * `geo_metrics_update_worker` - enabled at 1200 UTC - 12th Nov 2024 * No visible increase in errors that I can see ([Sidekiq Worker Detail chart for this worker](https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq3a-worker-detail?from=now-24h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gstg&var-environment=gstg&var-stage=main&var-worker=Geo::MetricsUpdateWorker)) * I'm also not seeing a huge variation in any of the patroni/sidekiq dashboard graphs. There was a deployment which happened just after this FF was turned on and these seem to normally cause a spike in metrics - not sure if there's a good way to filter out the noise caused by deploys? Maybe there's a way to get the query stats for just the Geo Metrics Worker by itself from grafana? * `geo_ci_secure_file_force_primary_checksumming` - enabled at 1400 UTC - 12th Nov 2024 * No visible increase in errors that I can see * `842` rows added to the `ci_secure_file_states` table (as expected since it's a 1-1 link to the source table) * Table size increase by `558kb` * [Checksumming seems to have completed](https://dashboards.gitlab.net/d/fdzkd1y2qpg5cf/checksumming?orgId=1) in \< 10 seconds given the small table size #### Phase 2 summary (WIP) * `geo_dependency_proxy_blob_force_primary_checksumming` - enabled at 1130 UTC - 13th Nov 2024 * No visible increase in errors that I can see * 1074 existing records updated * State table size increase by `795kb` to `860kb` * 7 verification failures (<1%) noted with the message `Error during verification: File is not checksummable` * `geo_dependency_proxy_manifest_force_primary_checksumming` - enabled at 1200 UTC - 13th Nov 2024 * No visible increase in errors that I can see * 74 existing records updated * State table size increase by `99kb` to `164kb` * `geo_group_wiki_repository_force_primary_checksumming` - enabled at 1235 UTC - 13th Nov 2024 * No visible increase in errors that I can see * 306 existing records updated * State table size increase by `115kb` to `262kb` * `geo_pages_deployment_force_primary_checksumming` - enabled at 1300 UTC - 13th Nov 2024 * No visible increase in errors that I can see * 27968 existing records updated * State table size increase by `12270kb` to `12400kb` * `geo_pipeline_artifact_force_primary_checksumming` - enabled at 1330 UTC - 13th Nov 2024 * No visible increase in errors that I can see * 37 existing records updated * State table size increase by `80kb` to `80kb` * `geo_terraform_state_version_force_primary_checksumming` - enabled at 1400 UTC - 13th Nov 2024 * No visible increase in errors that I can see * 22 existing records updated * State table size increase by `122.9kb` to `131kb` * 4 verification failures (18%) noted with the message `Error during verification: File is not checksummable` ### Grafana Screenshots <details> <summary>Checksumming</summary> geo_ci_secure_file_force_primary_checksumming  geo_dependency_proxy_blob_force_primary_checksumming  geo_dependency_proxy_manifest_force_primary_checksumming  geo_group_wiki_repository_force_primary_checksumming  geo_pages_deployment_force_primary_checksumming  geo_terraform_state_version_force_primary_checksumming  </details> <details> <summary>Patroni</summary> geo_ci_secure_file_force_primary_checksumming       geo_dependency_proxy_blob_force_primary_checksumming       geo_dependency_proxy_manifest_force_primary_checksumming       geo_group_wiki_repository_force_primary_checksumming       geo_pages_deployment_force_primary_checksumming       geo_terraform_state_version_force_primary_checksumming       </details> cc @mkozono @nsilva5 @luciezhao @sranasinghe @dbalexandre