Enforce the scheduled triggers setting

What does this MR do and why?

Step 2 of: #634139. MR 2/6.

Nothing read duo_scheduled_triggers_enabled, so turning it off had no effect. This MR enforces it at the three points a scheduled run can start.

Depends on !264600 (merged), which adds the setting.

Where Behaviour
Ai::RunFlowScheduleWorker#execute_flow Skips quietly and logs and returns without recording a failure
Ai::FlowTrigger#scheduled_event_type_available Rejects a newly added scheduled event type
Ai::FlowSchedules::CreateService Rejects the direct AiFlowScheduleCreate mutation

Design decisions

1. Wy does the worker skip quietly instead of failing?

  • Ai::FlowSchedule#record_failure! deactivates a schedule after MAX_CONSECUTIVE_FAILURES and emails owners. If a disabled setting counted as a failure, three ticks with the setting off would permanently deactivate the schedule and spam the owner.
  • And turning the setting back on would not restore it.
  • The skip mirrors the existing paused trigger branch in the worker, which exists for the same reason.

2. Why is the check in execute_flow and not validation_error?

  • advance_next_run_at! runs between the two (run_flow_schedule_worker.rb:32-34).
  • Checking in validation_error would freeze next_run_at while the setting is off, so re-enabling would fire every missed run at once.
  • Checking after the advance means re-enabling resumes at the next available tick, while preventing a catch up burst behaviour (see point 1 of "other thoughts").

3. Why does the trigger validation only block newly added schedule event types?

  • newly_added_scheduled_event_type? already existed for the feature flag case.
  • Reusing it means a trigger that already carries scheduled can still be saved for unrelated reasons even after the setting is turned off, for example, when it's deactivated, renamed, etc.
  • Without it, turning the setting off would make existing triggers unsaveable.

4. Why is FlowSchedules::UpdateService unguarded?

How to set up and validate locally

Prerequisites

  • Duo Enterprise enabled
  • GDK running with all services up
  • Self Managed Local Instance

1. Checkout this branch: 634139/sq/scheduled-triggers-enforcement

2. In Rails console

# 0. Set up

# Select project and group
project = Project.find(1000000) # or use your project ID
group = project.group

# Enable FFs
Feature.enable(:ai_flow_schedules, project)
Feature.enable(:autonomous_service_account_execution, project)

sfx = SecureRandom.hex(4)

item   = FactoryBot.create(:ai_catalog_flow, organization: project.organization, project: project)
parent = FactoryBot.create(:ai_catalog_item_consumer, :for_flow,
  group: group, project: nil, item: item)
consumer = FactoryBot.create(:ai_catalog_item_consumer, :for_flow,
  project: project, item: item, parent_item_consumer: parent)

sa = FactoryBot.create(:user, :service_account,
  username: "verif_sa_#{sfx}", organization: project.organization,
  composite_identity_enforced: false)

trigger = FactoryBot.create(:ai_flow_trigger,
  project: project, config_path: nil, user: nil,
  ai_catalog_item_consumer: consumer, autonomous_service_account: sa,
  event_types: [Ai::FlowTrigger::EVENT_TYPES[:scheduled]],
  description: "verification scheduled trigger #{sfx}")

schedule = FactoryBot.create(:ai_flow_schedule,
  flow_trigger: trigger, project: project, description: "verification schedule #{sfx}")

# Actual verification

# 1. Verify cascade and lock
project.duo_scheduled_triggers_enabled  # => true  (instance default)

project.project_setting.update!(duo_scheduled_triggers_enabled: false)
Project.find(project.id).duo_scheduled_triggers_enabled # => false

# group off, project inherits
project.project_setting.update!(duo_scheduled_triggers_enabled: nil)
group.namespace_settings.update!(duo_scheduled_triggers_enabled: false)
Project.find(project.id).duo_scheduled_triggers_enabled # => false

# lock rejects a project override
group.namespace_settings.update!(lock_duo_scheduled_triggers_enabled: true)

ps = ProjectSetting.find_by(project_id: project.id)
ps.duo_scheduled_triggers_enabled_locked? # => true
ps.duo_scheduled_triggers_enabled = true
ps.valid? # => false
ps.errors.full_messages # => ["Duo scheduled triggers enabled cannot be changed because it is locked by an ancestor"]

# 2. New scheduled triggers are rejected

group.namespace_settings.update!(lock_duo_scheduled_triggers_enabled: false)
project.project_setting.update!(duo_scheduled_triggers_enabled: false)

# create trigger
t = Ai::FlowTrigger.new(
  project: project, description: 'probe',
  config_path: '.gitlab/duo/probe.yaml',
  event_types: [Ai::FlowTrigger::EVENT_TYPES[:scheduled]]
)
t.valid?
t.errors[:event_types] # => ["scheduled events are turned off", ...]

# the trigger built in set up step 0 still saves for unrelated reasons
trigger.update(active: false) # => true
trigger.update(active: true) # => true

# 3. The worker skips quietly
schedule.update_column(:next_run_at, 1.minute.ago)

before    = Ai::FlowSchedule.find(schedule.id)
wf_before = Ai::DuoWorkflows::Workflow.where(trigger_flow_schedule_id: schedule.id).count

Ai::RunFlowScheduleWorker.new.perform(schedule.id, { 'scheduling' => true })

after = Ai::FlowSchedule.find(schedule.id)

before.next_run_at != after.next_run_at # => true advanced, so no backlog
after.consecutive_failure_count # => 0 no failure recorded
after.last_run_status # => nil status untouched
after.last_run_error # => nil
after.active  # => true not auto-deactivated
Ai::DuoWorkflows::Workflow.where(trigger_flow_schedule_id: schedule.id).count - wf_before # => 0

# 4. The direct mutation path is blocked
owner = group.owners.first&.user || User.find_by(admin: true)

Ai::FlowSchedules::CreateService.new(flow_trigger: trigger, current_user: owner)
  .execute(description: 'probe', cron: '0 9 * * 1', cron_timezone: 'UTC', active: true) # => success? false, message "Scheduled triggers are turned off for this project"

# 5. Re-enabling actually runs the flow
module RunServiceProbe
  class << self; attr_accessor :calls; end
  self.calls = 0

  def execute(*)
    RunServiceProbe.calls += 1
    ServiceResponse.success
  end
end
Ai::FlowTriggers::RunService.prepend(RunServiceProbe)

# With the setting off
schedule.update_column(:next_run_at, 1.minute.ago)
Ai::RunFlowScheduleWorker.new.perform(schedule.id, { 'scheduling' => true })

RunServiceProbe.calls # => 0
Ai::FlowSchedule.find(schedule.id).last_run_status # => nil

# With the setting on
project.project_setting.update!(duo_scheduled_triggers_enabled: true)
Project.find(project.id).duo_scheduled_triggers_enabled # => true

schedule.update_column(:next_run_at, 1.minute.ago)
Ai::RunFlowScheduleWorker.new.perform(schedule.id, { 'scheduling' => true })

on = Ai::FlowSchedule.find(schedule.id)

RunServiceProbe.calls # => 1
on.last_run_status # => "success"
on.consecutive_failure_count # => 0
on.next_run_at.future? # => true (one tick, not a queue of missed ones)

# 6. Cleanup

Groups::DestroyService.new(group, User.find_by(admin: true)).execute

MR Stack

  1. Add duo_scheduled_triggers_enabled cascading se... (!264600 - merged)
  2. This MR
  3. Add the scheduled automations setting to the Du... (!265569 - merged)
  4. Show when scheduled runs are turned off and blo... (!265545 - merged)
  5. Add scheduled automations setting to Projects (!266298 - merged)
  6. Exposes duo_scheduled_triggers_enabled in the p... (!267069)

References

Screenshots or screen recordings

Before After

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Shola Quadri

Merge request reports

Loading
Loading