Loading
Enforce the scheduled triggers setting
What does this MR do and why?
Step 2 of: #634139. MR 2/6.
Nothing read duo_scheduled_triggers_enabled, so turning it off had no effect. This MR enforces it at the three points a scheduled run can start.
Depends on !264600 (merged), which adds the setting.
| Where | Behaviour |
|---|---|
Ai::RunFlowScheduleWorker#execute_flow |
Skips quietly and logs and returns without recording a failure |
Ai::FlowTrigger#scheduled_event_type_available |
Rejects a newly added scheduled event type |
Ai::FlowSchedules::CreateService |
Rejects the direct AiFlowScheduleCreate mutation |
Design decisions
1. Wy does the worker skip quietly instead of failing?
Ai::FlowSchedule#record_failure!deactivates a schedule afterMAX_CONSECUTIVE_FAILURESand emails owners. If a disabled setting counted as a failure, three ticks with the setting off would permanently deactivate the schedule and spam the owner.- And turning the setting back on would not restore it.
- The skip mirrors the existing paused trigger branch in the worker, which exists for the same reason.
2. Why is the check in execute_flow and not validation_error?
advance_next_run_at!runs between the two (run_flow_schedule_worker.rb:32-34).- Checking in
validation_errorwould freeze next_run_at while the setting is off, so re-enabling would fire every missed run at once. - Checking after the advance means re-enabling resumes at the next available tick, while preventing a catch up burst behaviour (see point 1 of "other thoughts").
3. Why does the trigger validation only block newly added schedule event types?
newly_added_scheduled_event_type?already existed for the feature flag case.- Reusing it means a trigger that already carries scheduled can still be saved for unrelated reasons even after the setting is turned off, for example, when it's deactivated, renamed, etc.
- Without it, turning the setting off would make existing triggers unsaveable.
4. Why is FlowSchedules::UpdateService unguarded?
- The Product decision was to block creation.
How to set up and validate locally
Prerequisites
- Duo Enterprise enabled
- GDK running with all services up
- Self Managed Local Instance
1. Checkout this branch: 634139/sq/scheduled-triggers-enforcement
2. In Rails console
# 0. Set up
# Select project and group
project = Project.find(1000000) # or use your project ID
group = project.group
# Enable FFs
Feature.enable(:ai_flow_schedules, project)
Feature.enable(:autonomous_service_account_execution, project)
sfx = SecureRandom.hex(4)
item = FactoryBot.create(:ai_catalog_flow, organization: project.organization, project: project)
parent = FactoryBot.create(:ai_catalog_item_consumer, :for_flow,
group: group, project: nil, item: item)
consumer = FactoryBot.create(:ai_catalog_item_consumer, :for_flow,
project: project, item: item, parent_item_consumer: parent)
sa = FactoryBot.create(:user, :service_account,
username: "verif_sa_#{sfx}", organization: project.organization,
composite_identity_enforced: false)
trigger = FactoryBot.create(:ai_flow_trigger,
project: project, config_path: nil, user: nil,
ai_catalog_item_consumer: consumer, autonomous_service_account: sa,
event_types: [Ai::FlowTrigger::EVENT_TYPES[:scheduled]],
description: "verification scheduled trigger #{sfx}")
schedule = FactoryBot.create(:ai_flow_schedule,
flow_trigger: trigger, project: project, description: "verification schedule #{sfx}")
# Actual verification
# 1. Verify cascade and lock
project.duo_scheduled_triggers_enabled # => true (instance default)
project.project_setting.update!(duo_scheduled_triggers_enabled: false)
Project.find(project.id).duo_scheduled_triggers_enabled # => false
# group off, project inherits
project.project_setting.update!(duo_scheduled_triggers_enabled: nil)
group.namespace_settings.update!(duo_scheduled_triggers_enabled: false)
Project.find(project.id).duo_scheduled_triggers_enabled # => false
# lock rejects a project override
group.namespace_settings.update!(lock_duo_scheduled_triggers_enabled: true)
ps = ProjectSetting.find_by(project_id: project.id)
ps.duo_scheduled_triggers_enabled_locked? # => true
ps.duo_scheduled_triggers_enabled = true
ps.valid? # => false
ps.errors.full_messages # => ["Duo scheduled triggers enabled cannot be changed because it is locked by an ancestor"]
# 2. New scheduled triggers are rejected
group.namespace_settings.update!(lock_duo_scheduled_triggers_enabled: false)
project.project_setting.update!(duo_scheduled_triggers_enabled: false)
# create trigger
t = Ai::FlowTrigger.new(
project: project, description: 'probe',
config_path: '.gitlab/duo/probe.yaml',
event_types: [Ai::FlowTrigger::EVENT_TYPES[:scheduled]]
)
t.valid?
t.errors[:event_types] # => ["scheduled events are turned off", ...]
# the trigger built in set up step 0 still saves for unrelated reasons
trigger.update(active: false) # => true
trigger.update(active: true) # => true
# 3. The worker skips quietly
schedule.update_column(:next_run_at, 1.minute.ago)
before = Ai::FlowSchedule.find(schedule.id)
wf_before = Ai::DuoWorkflows::Workflow.where(trigger_flow_schedule_id: schedule.id).count
Ai::RunFlowScheduleWorker.new.perform(schedule.id, { 'scheduling' => true })
after = Ai::FlowSchedule.find(schedule.id)
before.next_run_at != after.next_run_at # => true advanced, so no backlog
after.consecutive_failure_count # => 0 no failure recorded
after.last_run_status # => nil status untouched
after.last_run_error # => nil
after.active # => true not auto-deactivated
Ai::DuoWorkflows::Workflow.where(trigger_flow_schedule_id: schedule.id).count - wf_before # => 0
# 4. The direct mutation path is blocked
owner = group.owners.first&.user || User.find_by(admin: true)
Ai::FlowSchedules::CreateService.new(flow_trigger: trigger, current_user: owner)
.execute(description: 'probe', cron: '0 9 * * 1', cron_timezone: 'UTC', active: true) # => success? false, message "Scheduled triggers are turned off for this project"
# 5. Re-enabling actually runs the flow
module RunServiceProbe
class << self; attr_accessor :calls; end
self.calls = 0
def execute(*)
RunServiceProbe.calls += 1
ServiceResponse.success
end
end
Ai::FlowTriggers::RunService.prepend(RunServiceProbe)
# With the setting off
schedule.update_column(:next_run_at, 1.minute.ago)
Ai::RunFlowScheduleWorker.new.perform(schedule.id, { 'scheduling' => true })
RunServiceProbe.calls # => 0
Ai::FlowSchedule.find(schedule.id).last_run_status # => nil
# With the setting on
project.project_setting.update!(duo_scheduled_triggers_enabled: true)
Project.find(project.id).duo_scheduled_triggers_enabled # => true
schedule.update_column(:next_run_at, 1.minute.ago)
Ai::RunFlowScheduleWorker.new.perform(schedule.id, { 'scheduling' => true })
on = Ai::FlowSchedule.find(schedule.id)
RunServiceProbe.calls # => 1
on.last_run_status # => "success"
on.consecutive_failure_count # => 0
on.next_run_at.future? # => true (one tick, not a queue of missed ones)
# 6. Cleanup
Groups::DestroyService.new(group, User.find_by(admin: true)).executeMR Stack
- Add duo_scheduled_triggers_enabled cascading se... (!264600 - merged)
- This MR
- Add the scheduled automations setting to the Du... (!265569 - merged)
- Show when scheduled runs are turned off and blo... (!265545 - merged)
- Add scheduled automations setting to Projects (!266298 - merged)
- Exposes duo_scheduled_triggers_enabled in the p... (!267069)
References
Screenshots or screen recordings
| Before | After |
|---|---|
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Shola Quadri