No Notifications for Failed Scheduled Pipelines
Summary
Scheduled pipelines, when created, are owned by that individual. However if that member leaves a project(we see this frequently as members switch teams etc.) then the scheduled pipeline fails to execute and the project team are non the wiser.
The fact the scheduled pipelines are expected to fail if the member no longer has access is not the issue, as it is called out in https://docs.gitlab.com/ee/user/project/pipelines/schedules.html#taking-ownership, however the lack of notifications for failed scheduled pipelines is the real problem.
Equally, scenarios such as an invalid .gitlab-ci.yml will cause pipelines to fail to be created and no notifications to be generated.
Steps to reproduce
- As a member of a project setup a scheduled pipeline
- Leave the project
Example Project
From #28718 (closed):
Errors generated when a scheduled pipeline with configuration errors are rescued without any thing logged which causes pipelines not to be triggered at all with no alert. In the case of scheduled specific jobs, this may not necessarily be caught at any other point
See https://gitlab.com/gitlab-org/gitlab-ee/issues/26893#note_215674214 for an example of this happening
def run_pipeline_schedule(schedule, user)
Ci::CreatePipelineService.new(schedule.project,
user,
ref: schedule.ref)
.execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule)
rescue Ci::CreatePipelineService::CreateError
# no-op. This is a user operation error such as corrupted .gitlab-ci.yml.
rescue => e
error(schedule, e)
end
What is the current bug behavior?
No notifications are received for the failure and there is no entry in the pipelines tab for these failed scheduled pipelines
What is the expected correct behavior?
If a scheduled pipeline fails to execute either each project member or maintainers/owners at least, should receive some notification(email?) that it failed to execute.
Relevant logs and/or screenshots
N/A
Output of checks
This bug happens on GitLab.com
Results of GitLab environment info
N/A
Results of GitLab application Check
N/A
Possible fixes
- Send an email to maintainer(s) for the project - Send an email, the status of the pipeline will be "Failed to Create pipeline". This will match the status shown on the Scheduled Pipelines interface when that issue is implemented.
Add a todo for the schedule owner with the failure-
Add to the pipeline page with a failed pipelineWe deemed this too noisy for now, the pipeline page is already hard to understand.
Proposal
We would need to check if the response was persisted in run_pipeline_schedule_worker.rb and if it's failed generate an email template (see attached) similar to how current templates are done here: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/mailers/emails/pipelines.rb
This would be applicable to pipelines that failed due to invalid permissions. For pipelines that failed to to invalid ci, there's already an email template sent out. We can enhance the current template with the mockups attached.
This change should be Feature flagged as it'll cause increase in noise/emails. Should pull some metrics to see scope of impact