Pipeline Observability First Iteration

added devopssystems group::distributionbuild groupdistribution sectioncore platform spike labels

changed title from Pipeline Observability to Pipeline Observability First Iteration

Thanks for writing this up @rmarshall

Reduced awareness of pipeline failures outside of triage because they are not alerted in Slack

At the moment, all Slack notifications are still active - none have been disabled. @twk3 suggested modifying the notifications to link to the generated issue, rather than eventually disabling the notifications.

Finding pipeline issues is more difficult because they are spread across multiple projects

Definitely true.

As far as I know, we're the only team using this project at the moment, so a simple label filter on gitlab-org/ shows the aggregated list.

Issue bot also supports custom labels, so we could always update our CI configuration to include ~"group::distribution" so we have a reliable filter for all Distribution-owned failures.

If an approach like this sounds feasible, the benefit is not having to write and maintain more code to create a separate/custom dashboard for monitoring these issues. Open to any thoughts here.

what's missing from above is the intent to have notification whenever certain threshold of "stale" pipeline failure issues have been reached. In other words - alerting akin to what we have in MR alerts - informing whenever we breach time threshold on MR assignment/response.

While I agree that what you ( @mnielsen ) propose is a "less impact" approach, I can see it as a first iteration but not necessarily as a final implementation due to what I've outlined above.

I think @dmakovey and I are in a bit of agreement here on reporting a certain threshold of pipeline failureneeds investigation issues. The stale issues are a good thing to know but it would also be a good thing to know if the number of failure have jumped and everyone is busy and not paying attention.

I don't know what that threshold should be. Maybe 6 issues across all the projects? Anyone think that this is too high or too low? Do we need to put a time interval there so that stale issues are not being counted?

I've created a Distribution Pipeline Failures issue board at the gitlab-org/ group level so we can see all pipeline failures across our projects at once.

As mentioned, if others adopt this tool, we'll need to tell issue-bot to also apply groupdistribution to created issues to ensure this board is filtered to only issues relevant to us.

what's missing from above is the intent to have notification whenever certain threshold of "stale" ~"pipeline failure" issues have been reached. In other words - alerting akin to what we have in MR alerts - informing whenever we breach time threshold on MR assignment/response.

Do we have a definition for what "stale" would mean here? For MRs, we have a 3-day SLO. We don't have an SLO for pipelines as far as I know. At the moment, issues are only created for critical branches like master and stable-*, so technically speaking those should take immediate priority to address master:broken as soon as possible.

If a pipeline is fixed, the issue should be closed.
If a root cause needs investigation, a root cause issue should be created.

So really, issue-bot-generated issues really shouldn't live long enough to get stale. If they do, we'll simply need to put more focus/priority on closing them given the importance of having passing pipelines on default branches.

In terms of thresholds, it should be fairly easy to see a spike in open issues by looking at the board I linked above. What do you think?

Given the temporal nature of pipeline issues, I'd say that any pipeline issue open more than 72 hours without being closed into a root cause issue or a spike is "stale" and should be in red status.

given @mnielsen and @rmarshall we shall implement time-based (vs load based) threshold for pipeline failure issues and alert whenever we have issues in pipeline failureneeds investigation state for longer than 72h?

mentioned in commit monitoring@51d9e0b3

mentioned in merge request monitoring!17 (merged)

I have opened a naive draft as a starting point of discussion.

The slack notification piece will come next, but this gives us something more concrete to discuss:

monitoring!17 (merged)

mentioned in issue #1022 (closed)

I am working on a pipelines dashboard for monitoring#4 (closed), but that is more for historical data, tracking pipeline failures and durations over time. The proposal here is more about being notified of master/main failures, so I don't see much overlap.