Introduce alert deprecation notice on metrics dashboard

changed milestone to %13.12

@crystalpoole - here's the issue for adding the deprecation alert on the metrics page. I will draft some text and run it by TW to make sure it's okay!

cc: @tristan.read, @oregand and @ohoral in case any of you have capacity to pick this up in 13.12. Guessing we should probably get it up there as soon as possible.

Side questions for y'all - is it possible to only display the alert on those dashboards where managed prometheus is being utilized (meaning, only on those dashboards where the functionality was previously available)? Also, will we have to create an issue to remove this alert when the deprecation is complete?

This seems fairly important so I can jump on this right away.

I'll assume we're are going ahead with the plan to deprecate in %14.0 but I'll keep an eye on the discussion in #327796 (comment 558372822) in case it gets delayed.

cc. @crystalpoole

is it possible to only display the alert on those dashboards where managed prometheus is being utilized

@ameliabauerly I haven't confirmed, but my gut feel is yes - this should be possible. We already have managed-prometheus-specific functionality where we show/hide the alert item in the dropdown, we should be able to use the same mechanism for displaying the deprecation notice.

Also, will we have to create an issue to remove this alert when the deprecation is complete?

Yes we should add something to the %14.0 plan, and track in the overall issue #327796 (closed). There will be other things to track for %14.0 too - e.g. updating the docs about the removal of the functionality.

Cool - created! Adding it to the 14.0 planning issue and to #327796 (closed) as well

marked this issue as related to #327796 (closed)

mentioned in issue #327796 (closed)

changed the description

assigned to @tristan.read

added Category:Metrics deprecation frontend labels

Current alert on the docs page:

Proposed new alert for the metrics page:

GitLab-managed Prometheus has been deprecated and is scheduled for removal. As part of this deprecation, all alerting functionality on the Metrics dashboard will no longer be available. If you are currently using alerts on the metrics dashboard, your alerts will be migrated. But you will no longer be able to add or edit existing alerts from this dashboard.

@ngaskill - this copy feels like a bit of a mess, could you help? Basically, a bunch of features are going to disappear from the metrics dashboard. We want to let people know that's going to happen and what the impact will be.

@tristan.read and @seanarnold - is my understanding of what will happen here correct? Meaning, that existing alerts will be migrated but users will no longer be able to add or edit existing alerts?

The question is if we want to continue to alert on these alerts or not. It may be cleaner to stop responding to these alerts as without the ability to edit or delete the alerts we are left with half-baked functionality.

@kbychu had some thoughts on this I think

Viktor added some context on this on #327796 (comment 558617653):

As far as I know the impact is only about configuring alerts. Already configured alerts will continue to work once #327655 (closed) is done.

From this it sounds like the current plan is to continue to alert on already configured alerts, and I think this is probably the safest way forward. I definitely agree it leaves a half-baked experience for those users but, I think it's probably more risky for those alerts to just disappear, given the concerns @syasonik and @tristan.read raised in #327796 (comment 558372822):

One month between the deprecation notice on the product, and Alerts becoming non-configurable (and possibly not firing at all? I'm not clear on that part) seems quite aggressive for users where these alerts make up a critical part of their organisation.

@ameliabauerly Here's a quick edit:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, all alerting functionality on the Metrics dashboard will no longer be available. If you currently use alerts on the Metrics dashboard, your alerts will be migrated. However, you will no longer be able to add or edit existing alerts from this dashboard.

One question: It says that existing alerts will be migrated. Migrated to where?

Thanks so much, @ngaskill!

One question: It says that existing alerts will be migrated. Migrated to where?

This is an excellent question, and an important detail to include! The migration issue says they are being migrated to the new Prometheus cluster integration. How would you recommend we update the text to accomodate?

@ameliabauerly I'm a bit confused on the status of alerts here. The docs for Prometheus cluster integration say that alerts are not currently supported. I took a look at the migration issue, and there is one comment saying that alerts should be migrated, but the rest of the issue just talks about removing them.

@ngaskill - do you mean this migration issue: #327655 (closed)? It seems like this one focuses on migrating rather than removing? Or are you meaning something else?

When you say status of alerts, I realized there might be some confusion between the alerts created in the metrics dashboard UI (which is functionality that is only available when someone has GitLab-managed Prometheus enabled) and alerts more generally, such as those displayed on Operations > Alerts.

When everyone is talking about removing the alerts in, for example, in &5877, they are talking about removing the existing UI for adding and editing alerts from the metrics dashboard only. So, in the future, people will not be able to add new alerts or edit existing alerts on the metrics dashboard - all that functionality will be gone. I'm guessing that's why the docs say that alerts are not currently supported because the functionality to support them (GitLab-managed Prometheus) is being removed.

However, I believe the plan is that the alerts that had previously been created from the metrics dashboard will continue alerting (meaning, they will continue to fire and appear in Operations > Alerts) so that people who relied on those alerts coming from the metrics dashboard previously don't suddenly have their alerts disappear. That's what the migration issue is for, at least as far as I know.

But alerts more generally - alerts coming in via integrations set up in Settings > Operations and displayed on Operations > Alerts will still continue to exist. None of that functionality is being impacted at all.

Does that help at all, @ngaskill? If not, happy to have a call if it's helpful to talk this through! (@tristan.read, @seanarnold, also if I'm getting any of that wrong, definitely feel free to correct!)

@ameliabauerly Thanks for the detail. I was referring to the alerts for the metrics dashboard UI. If the plan is to move those alerts to Operations > Alerts, then we can add that to the copy:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, all alerting functionality on the Metrics dashboard will no longer be available. If you currently use alerts on the Metrics dashboard, your alerts will be migrated to Operations > Alerts. However, you will no longer be able to add or edit existing alerts from the Metrics dashboard.

Thanks, @ngaskill! With this sentence:

If you currently use alerts on the Metrics dashboard, your alerts will be migrated to Operations > Alerts.

It's not that the alerts will be migrated to Operations > Alerts; it's more that we're migrating how the alerts are being processed - from GitLab-managed Prometheus to the new cluster integration. The alerts under discussion already showed up on the Alert list page and they will (hopefully) continue to do so. But, we're still trying to figure out what will be migrated, and who will be doing the migrating (us or if we're going to ask users to do it themselves), which makes finalizing the text here a little tricky.

@crystalpoole, @kbychu, since we don't yet know whether we're going to migrate the alerts automatically or make users do that manually - I'm not sure what we should put in our deprecation alert. If we are migrating the alerts ourselves, we would say something like:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, all alerting functionality on the Metrics dashboard will no longer be available. If you currently use alerts on the Metrics dashboard, your alerts will be migrated to the new cluster integration automatically. However, you will no longer be able to add or edit existing alerts from this dashboard.

If we are expecting users to migrate the alerts themselves, we would instead say something like:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, all alerting functionality on the Metrics dashboard will no longer be available. If you currently use alerts on the Metrics dashboard, you must migrate your alerts [following these instructions link to docs]. You will no longer be able to add or edit existing alerts from this dashboard.

It seems like it's important to surface what's actually happening to these alerts to users in this message (especially if they are needing to take action to prevent their alerts from disappearing). But, until we know which route we're going in, we won't really be able to finalize this text, or get this message up on the site.

Do we have a sense of when we'll be able to make that call, of whether we're migrating the alerts ourselves or whether we're going to ask users to do it? If we won't be able to make it soon, should we try to come up with a more generic message? Something like:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, all alerting functionality on the Metrics dashboard will no longer be available, meaning you will no longer be able to add or edit existing alerts from this dashboard.

This message seems less helpful overall, but I suppose we could get it up faster. WDYT? Would appreciate your thoughts! Thank you :)

Just let me know once we decide how/to where the migration is happening, and I can review the text

@ameliabauerly @seanarnold

Chiming in way late here

I think it's probably more risky for those alerts to just disappear

Agreed, this buys the user time to move their workflow to somewhere else

mentioned in issue #328775 (closed)

I think there's two places we could add a deprecation notice:

Agree, I think this could be a nice touch, @seanarnold! I've added that detail to this issue's description :)

changed the description

mentioned in epic &5877

added to epic &5877

changed the description

mentioned in issue #328857 (closed)

added Deliverable workflowready for development labels

added sectionops label

@ngaskill - starting a fresh thread since the other one has gotten very long We've just decided that we are migrating alerts for users (rather than having them do it themselves). Kevin suggested the following text for the metrics dashboard:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, we will migrate your existing alerts to the new cluster integration and they will continue to function. However, you will no longer be able to add or edit existing alerts from this dashboard.

I suppose we'd need a slight variation on this text for the alerts list page. Maybe:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, we will migrate your existing alerts to the new cluster integration and they will continue to function. However, you will no longer be able to add or edit existing alerts from the Metrics page.

Thoughts on this text? Thank you!

cc: @tristan.read

@ameliabauerly I don't think we need to specifically mention that we migrated the alerts, as long as we tell the user where to find them. Here's a slight edit:

GitLab-managed Prometheus is deprecated and scheduled for removal. Following this removal, your existing alerts will continue to function as part of the new cluster integration. However, you will no longer be able to add new alerts or edit existing alerts from the metrics dashboard.