<!-- Implementation issues are used break-up a large piece of work into small, discrete tasks that can move independently through the build workflow steps. They're typically used to populate a Feature Epic. Once created, an implementation issue is usually refined in order to populate and review the implementation plan and weight. Example workflow: https://about.gitlab.com/handbook/engineering/development/threat-management/planning/diagram.html#plan --> ## Why are we doing this work <!-- A brief explanation of the why, not the what or how. Assume the reader doesn't know the background and won't have time to dig-up information from comment threads. --> To support scheduled scans configured in Security Orchestration Policies we need to add workers and services to configure and execute scheduled jobs with selected scans. ## Relevant links - [Epic](https://gitlab.com/groups/gitlab-org/-/epics/5360) ## Non-functional requirements <!-- Add details for required items and delete others. --> - [-] Documentation: added in separate issue, - [x] Feature flag: all work will be hidden behind already created feature flag: `security_orchestration_policies_configuration` - [-] Performance: verify the impact of additional cron jobs on the system, coordinate with `CI` team, - [x] Testing: - [x] verify if `pipeline` scan policies are working properly, - [x] verify if `schedule` scan policies are working properly: - [x] check multiple cron expressions (with `*`, `SUN,MON`, numbers, etc.) - [x] check if jobs are properly scheduled when the repository with policies is updated, ## Implementation plan <!-- Steps and the parts of the code that will need to get updated. The plan can also call-out responsibilities for other team members or teams. --> - [x] ~backend extend `config/initializers/1_settings.rb` with `Security::CreateOrchestrationPolicyWorker` and `Security::OrchestrationPolicyRuleScheduleWorker`, properly set `cron` value for these jobs, `CreateOrchestrationPolicyWorker` can work every hour, but `OrchestrationPolicyRuleScheduleWorker` must work quite often (ie. every 15 minutes) to make sure we can run scheduled scans on provided by users schedule (mention that in the documentation), - [x] ~backend extend `ee/app/models/security/orchestration_policy_configuration.rb` with scope `:with_outdated_configuration` that will get all policy configurations that does not have value for `configured_at` field or for which value of `configured_at` is earlier than `security_policy_management_project.last_activity_at` -> this is not a perfect way to get information when was the last push to `security_policy_management_project` (and potential policy update), but it will significantly reduce calls to database to create `OrchestrationPolicyRuleSchedule` in `ee/app/workers/security/create_orchestration_policy_worker.rb`, - [x] ~backend extend `ee/app/models/security/orchestration_policy_rule_schedule.rb` with scope `runnable_schedules` and methods needed by `Schedulable` concern: `ideal_next_run_from`, `cron_worker_next_run_from` and `set_next_run_at`; `policy` method (to load policy configuration), - [x] ~backend add `ee/app/workers/security/create_orchestration_policy_worker.rb` that will go through `Security::OrchestrationPolicyConfiguration.with_outdated_configuration` in batches, and for each outdated configuration it will call `Security::OrchestrationPolicies::ProcessRuleService` service, - [x] ~backend add `ee/app/services/security/orchestration_policies/process_rule_service.rb` that will recreate `Security::OrchestrationPolicyRuleSchedule` in database for each `schedule` policy and set `configured_at` for `OrchestrationPolicyConfiguration`, - [x] ~backend add `ee/app/workers/security/orchestration_policy_rule_schedule_worker.rb` that will go through `Security::OrchestrationPolicyRuleSchedule.runnable_schedules` (all schedules with `next_run_at < Time.now`) in batches, and for each runnable schedule it will call `Security::OrchestrationPolicies::RuleScheduleService` service, - [x] ~backend add `ee/app/services/security/orchestration_policies/rule_schedule_service.rb` that will set new value for `next_run_at` field and go through actions in provided policy and start new pipeline with configured DAST On Demand Scan using `::DastOnDemandScans::CreateService`, when triggering a pipeline take last user that edited the policy file, <!-- Workflow and other relevant labels ~"group::" ~"Category:" ~"GitLab Ultimate" --> <!-- Other settings you might want to include when creating the issue. -->