Skip to content

Adds sidekiq workers to cascade ai settings

Surabhi Sumanrequested to merge
ss/cascade-duo-remote-flows-setting into master

What does this MR do and why?

This adds sidekiq workers to cascade application_settings and namespace_settings when they are updated via application code.

Currently if project level setting is nil it will inherit the parent value. If the project ever has it set to true or false, however, it will no longer inherit the value from the parent. For users, it is impossible to tell if the child value is nil or just inheriting so this behaviour is a bit confusing and unexpected. It also has negative performance consequences to always be traversing up the hierarchy to find values. We add a sidekiq worker so that when a new default value is set on a parent, we re-set all children to have that value. This is further explained in the dev docs: https://docs.gitlab.com/development/cascading_settings/#cascading-setting-values-on-write

From comment !202745 (comment 2725414538)

Scope of change

This MR adds two new generic workers to update any duo cascading settings as we have a couple of cascading settings that need to be handled similarly and the existing CascadeDuoDeaturesEnabledWorker were too specific.

I've kept existing ee/app/workers/app_config/cascade_duo_features_enabled_worker.rb and ee/app/workers/namespaces/cascade_duo_features_enabled_worker.rb to handle compatibility for already scheduled workers. Going forward, duo_features_enabled setting will be using the generic workers too and we can cleanup these afterwards.

Performance discussion

Detailed discussion in #442164 (comment 2756758950)

Conclusion and Alignment

We agreed with MR 204464 implementing the mass update approach to maintain consistency between existing and new Duo settings, while acknowledging that the broader cascading settings framework performance issue (issue #442164) remains open for future architectural improvements. Detailed discussion in #442164 (comment 2756758950)

Database

Database analysis of queries in the newly-added Worker:

Group#all_projects

For finding all projects within the root group and its descendants.

SQL:

puts Group.first.all_projects.to_sql

SELECT "projects"."id", "projects"."name", "projects"."path", "projects"."description", "projects"."created_at", "projects"."updated_at", "projects"."creator_id", "projects"."namespace_id", "projects"."last_activity_at", "projects"."import_url", "projects"."visibility_level", "projects"."archived", "projects"."avatar", "projects"."merge_requests_template", "projects"."star_count", "projects"."merge_requests_rebase_enabled", "projects"."import_type", "projects"."import_source", "projects"."approvals_before_merge", "projects"."reset_approvals_on_push", "projects"."merge_requests_ff_only_enabled", "projects"."issues_template", "projects"."mirror", "projects"."mirror_last_update_at", "projects"."mirror_last_successful_update_at", "projects"."mirror_user_id", "projects"."shared_runners_enabled", "projects"."runners_token", "projects"."build_allow_git_fetch", "projects"."build_timeout", "projects"."mirror_trigger_builds", "projects"."pending_delete", "projects"."public_builds", "projects"."last_repository_check_failed", "projects"."last_repository_check_at", "projects"."only_allow_merge_if_pipeline_succeeds", "projects"."has_external_issue_tracker", "projects"."repository_storage", "projects"."repository_read_only", "projects"."request_access_enabled", "projects"."has_external_wiki", "projects"."ci_config_path", "projects"."lfs_enabled", "projects"."description_html", "projects"."only_allow_merge_if_all_discussions_are_resolved", "projects"."repository_size_limit", "projects"."printing_merge_request_link_enabled", "projects"."auto_cancel_pending_pipelines", "projects"."service_desk_enabled", "projects"."cached_markdown_version", "projects"."delete_error", "projects"."last_repository_updated_at", "projects"."disable_overriding_approvers_per_merge_request", "projects"."storage_version", "projects"."resolve_outdated_diff_discussions", "projects"."remote_mirror_available_overridden", "projects"."only_mirror_protected_branches", "projects"."pull_mirror_available_overridden", "projects"."jobs_cache_index", "projects"."external_authorization_classification_label", "projects"."mirror_overwrites_diverged_branches", "projects"."pages_https_only", "projects"."external_webhook_token", "projects"."packages_enabled", "projects"."merge_requests_author_approval", "projects"."pool_repository_id", "projects"."runners_token_encrypted", "projects"."bfg_object_map", "projects"."detected_repository_languages", "projects"."merge_requests_disable_committers_approval", "projects"."require_password_to_approve", "projects"."max_pages_size", "projects"."max_artifacts_size", "projects"."pull_mirror_branch_prefix", "projects"."remove_source_branch_after_merge", "projects"."marked_for_deletion_at", "projects"."marked_for_deletion_by_user_id", "projects"."autoclose_referenced_issues", "projects"."suggestion_commit_message", "projects"."project_namespace_id", "projects"."hidden", "projects"."organization_id" FROM "projects" WHERE "projects"."namespace_id" IN (SELECT namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] AS id FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND (traversal_ids @> ('{22}')))

Postgres.ai analysis for this sql using the gitlab-org root group:

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/43957/commands/134489

Group#self_and_descendants

For finding all groups within the root group.

SQL:

puts Group.first.self_and_descendants.to_sql

SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND (traversal_ids @> ('{22}'))

Postgres.ai analysis for this sql using the gitlab-org root group: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/43957/commands/134490

NamespaceSetting.for_namespaces(namespace_ids)

For finding all namespace settings with duo settings set to a certain value for a set of namespaces.


SELECT "namespace_settings"."created_at", "namespace_settings"."updated_at", "namespace_settings"."namespace_id", "namespace_settings"."prevent_forking_outside_group", "namespace_settings"."allow_mfa_for_subgroups", "namespace_settings"."default_branch_name", "namespace_settings"."repository_read_only", "namespace_settings"."resource_access_token_creation_allowed", "namespace_settings"."prevent_sharing_groups_outside_hierarchy", "namespace_settings"."new_user_signups_cap", "namespace_settings"."setup_for_company", "namespace_settings"."jobs_to_be_done", "namespace_settings"."runner_token_expiration_interval", "namespace_settings"."subgroup_runner_token_expiration_interval", "namespace_settings"."project_runner_token_expiration_interval", "namespace_settings"."show_diff_preview_in_email", "namespace_settings"."enabled_git_access_protocol", "namespace_settings"."unique_project_download_limit", "namespace_settings"."unique_project_download_limit_interval_in_seconds", "namespace_settings"."unique_project_download_limit_allowlist", "namespace_settings"."auto_ban_user_on_excessive_projects_download", "namespace_settings"."only_allow_merge_if_pipeline_succeeds", "namespace_settings"."allow_merge_on_skipped_pipeline", "namespace_settings"."only_allow_merge_if_all_discussions_are_resolved", "namespace_settings"."default_compliance_framework_id", "namespace_settings"."runner_registration_enabled", "namespace_settings"."allow_runner_registration_token", "namespace_settings"."unique_project_download_limit_alertlist",  "namespace_settings"."experiment_features_enabled", "namespace_settings"."default_branch_protection_defaults", "namespace_settings"."service_access_tokens_expiration_enforced", "namespace_settings"."product_analytics_enabled", "namespace_settings"."allow_merge_without_pipeline", "namespace_settings"."enforce_ssh_certificates", "namespace_settings"."math_rendering_limits_enabled", "namespace_settings"."lock_math_rendering_limits_enabled", "namespace_settings"."duo_features_enabled", "namespace_settings"."lock_duo_features_enabled",  "namespace_settings"."duo_remote_flows_enabled", "namespace_settings"."lock_duo_remote_flows_enabled", "namespace_settings"."disable_personal_access_tokens", "namespace_settings"."enable_auto_assign_gitlab_duo_pro_seats" FROM "namespace_settings" WHERE "namespace_settings"."namespace_id" IN (125, 126) AND "namespace_settings"."duo_features_enabled" = FALSE AND "namespace_settings"."duo_remote_flows_enabled" = FALSE

Postgres.ai analysis for this sql using the gitlab-org root group:

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/43957/commands/134491

ProjectSetting.for_projects(project_ids_to_update)

For finding all project settings with duo_feature set to a certain value for a set of projects.

SELECT
    "project_settings"."project_id",
    "project_settings"."created_at",
    "project_settings"."updated_at",
    "project_settings"."push_rule_id",
    "project_settings"."show_default_award_emojis",
    "project_settings"."allow_merge_on_skipped_pipeline",
    "project_settings"."squash_option",
    "project_settings"."has_confluence",
    "project_settings"."has_vulnerabilities",
    "project_settings"."prevent_merge_without_jira_issue",
    "project_settings"."cve_id_request_enabled",
    "project_settings"."mr_default_target_self",
    "project_settings"."previous_default_branch",
    "project_settings"."warn_about_potentially_unwanted_characters",
    "project_settings"."merge_commit_template",
    "project_settings"."has_shimo",
    "project_settings"."squash_commit_template",
    "project_settings"."legacy_open_source_license_available",
    "project_settings"."target_platforms",
    "project_settings"."enforce_auth_checks_on_uploads",
    "project_settings"."selective_code_owner_removals",
    "project_settings"."issue_branch_template",
    "project_settings"."show_diff_preview_in_email",
    "project_settings"."suggested_reviewers_enabled",
    "project_settings"."only_allow_merge_if_all_status_checks_passed",
    "project_settings"."mirror_branch_regex",
    "project_settings"."allow_pipeline_trigger_approve_deployment",
    "project_settings"."emails_enabled",
    "project_settings"."pages_unique_domain_enabled",
    "project_settings"."pages_unique_domain",
    "project_settings"."runner_registration_enabled",
    "project_settings"."product_analytics_instrumentation_key",
    "project_settings"."product_analytics_data_collector_host",
    "project_settings"."cube_api_base_url",
    "project_settings"."encrypted_cube_api_key",
    "project_settings"."encrypted_cube_api_key_iv",
    "project_settings"."encrypted_product_analytics_configurator_connection_string",
    "project_settings"."encrypted_product_analytics_configurator_connection_string_iv",
    "project_settings"."pages_multiple_versions_enabled",
    "project_settings"."allow_merge_without_pipeline",
    "project_settings"."duo_features_enabled",
    "project_settings"."duo_remote_flows_enabled",
    "project_settings"."require_reauthentication_to_approve"
FROM
    "project_settings"
WHERE
    "project_settings"."project_id" = 41 AND
    "project_settings"."duo_features_enabled" = FALSE AND
    "project_settings"."duo_remote_flows_enabled" = FALSE;

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/43957/commands/134492

References

Related to #569137

Screenshots or screen recordings

Before After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Surabhi Suman

Merge request reports