Skip to content

No Notifications for Failed Scheduled Pipelines

Summary

Scheduled pipelines, when created, are owned by that individual. However if that member leaves a project(we see this frequently as members switch teams etc.) then the scheduled pipeline fails to execute and the project team are non the wiser.

The fact the scheduled pipelines are expected to fail if the member no longer has access is not the issue, as it is called out in https://docs.gitlab.com/ee/user/project/pipelines/schedules.html#taking-ownership, however the lack of notifications for failed scheduled pipelines is the real problem.

Equally, scenarios such as an invalid .gitlab-ci.yml will cause pipelines to fail to be created and no notifications to be generated.

Steps to reproduce

  • As a member of a project setup a scheduled pipeline
  • Leave the project

Example Project

From #28718 (closed):

Errors generated when a scheduled pipeline with configuration errors are rescued without any thing logged which causes pipelines not to be triggered at all with no alert. In the case of scheduled specific jobs, this may not necessarily be caught at any other point

See https://gitlab.com/gitlab-org/gitlab-ee/issues/26893#note_215674214 for an example of this happening

  def run_pipeline_schedule(schedule, user)
    Ci::CreatePipelineService.new(schedule.project,
                                  user,
                                  ref: schedule.ref)
      .execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule)
  rescue Ci::CreatePipelineService::CreateError
    # no-op. This is a user operation error such as corrupted .gitlab-ci.yml.
  rescue => e
    error(schedule, e)
  end

What is the current bug behavior?

No notifications are received for the failure and there is no entry in the pipelines tab for these failed scheduled pipelines

What is the expected correct behavior?

If a scheduled pipeline fails to execute either each project member or maintainers/owners at least, should receive some notification(email?) that it failed to execute.

Relevant logs and/or screenshots

N/A

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

N/A

Results of GitLab application Check

N/A

Possible fixes

  • Send an email to maintainer(s) for the project - Send an email, the status of the pipeline will be "Failed to Create pipeline". This will match the status shown on the Scheduled Pipelines interface when that issue is implemented.
  • Add a todo for the schedule owner with the failure
  • Add to the pipeline page with a failed pipeline We deemed this too noisy for now, the pipeline page is already hard to understand.

Implementation Guide

Proposal

Updated proposal:

  1. Send a single notification to project owner when a schedule owner has been:
    1. deleted
    2. leaves the project
    3. is disabled
  2. The following will be the subject of the email: Assign a new owner to the pipeline schedule: [Pipeline schedule name]
  3. Copy for the email
    1. The owner of the pipeline schedule [Pipeline schedule name] is no longer available. Without an owner, this schedule will fail to run. Assign a new owner to ensure the pipeline runs as expected.
    2. Visual: schedule-pipeline-notification.png
  4. Initial requirements are for 1 email per project only and no further reminders. #36806 (comment 2115229398)
OLD

We would need to check if the response was persisted in run_pipeline_schedule_worker.rb and if it's failed generate an email template (see attached) similar to how current templates are done here: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/mailers/emails/pipelines.rb

This would be applicable to pipelines that failed due to invalid permissions. For pipelines that failed to to invalid ci, there's already an email template sent out. We can enhance the current template with the mockups attached.This change should be Feature flagged as it'll cause increase in noise/emails. Should pull some metrics to see scope of impact

Edited by 🤖 GitLab Bot 🤖