Add alert definition to source controlled dashboards
Problem to solve
We should be able to add alert definition to source controlled custom dashboards in the same way that alerts are defined in default dashboards.
Intended users
Further details
Proposal
Permissions and Security
Documentation
Testing
What does success look like, and how can we measure that?
Links / references
Designs
Relates to
- #5997412.1
- gitlab #2824113.3
Activity
added typefeature label
mentioned in issue #59974 (closed)
marked this issue as related to #59974 (closed)
added devopsmonitor + 1 deleted label
changed milestone to %12.0
@sengelhard - Can you point me to someone on the team I could discuss this with? My key question is if "in dashboard" is the only way to source-control the alert definition. I'd like to have that alert definition happen in source control but I'd rather not start with that being tied to the "Create a dashboard" workflow.
@kencjohnston I don't think alert definitions would need to be in the dashboard definition, though that would certainly simplify the process. I think another good question is whether we'd want to allow users to define computed alerts in source control (as brought up in https://gitlab.com/gitlab-org/gitlab-ee/issues/5456#note_174671249).
Right now, there are a few things the back end needs to set an alert:
- which metric(s?)
- which environment
- which project
- details of the alert
How alerts and metrics work now: A metric in a yml-defined dashboard is given an identifier that the author has determined to be unique. We use that identifier to associate a yml-defined metric with a metric database record. A metric database record is required for alert to be created. Key detail: Source controlled dashboards don't yet support alerts set in the UI either - see #60905 (moved) and #60907 (moved). So that work would be prerequisite to this.
Back to your original question:
We could separate out alert definitions into its own file and it could certainly work, but the costs would be engineering time, some intuitiveness, and probably some user time redefining/referencing metric info. If we allow alerts computed based on multiple metrics, I think each of those costs go up.
We may also need to consider gitlab-defined metrics such as cluster metrics or the dashboard metrics we have available now. If we want users to be able to set alerts for those metrics in source control, the definitions would need to be separated from the "create a dashboard" workflow, as the users aren't creating those dashboards at all.
From a UX perspective, I think we want to be careful with this entire ticket. The expected source of truth & what reasonable behavior looks like for our users could get messy quickly. For example: what happens if a user deletes a source-control-defined alert in the UI? Does the alert get recreated on the next pipeline build? Do we open an MR with the alert deletion from source? Do we let the discrepancy live? Which source of info do we trust?
I'm happy to chat more about this - just let me know!
Thanks @syasonik - this was super helpful. I'm not feeling like we have enough proposal here, and certainly a ton of questions to consider this for 12.1 so I'm pushing it out, but I would like to continue the discussion with you. I'll see if I can find time on your calendar for a call that we can ensure we record and invite others to. Thanks!
-
@kencjohnston Context for source-controlled dashboards: https://gitlab.com/gitlab-org/gitlab-ce/issues/59974
-
Recapping the discussion from our call:
- Current state of the world: Metrics (representing a prometheus query & everything we need to display the results of the query) are managed from many locations - DB (via UI), DB (via GitLab migrations), and a project’s repo. Each of these avenues functions differently.
- Raised high-level questions:
- Is going down the path of source-controlling our figuration the direction we want to go right now?
- Do users find the source-controlled dashboards useful?
- If we go down this path, how do we want to handle establishing source control as the source of truth?
To see notes, reference the google doc.
changed milestone to %12.1
added APM label
marked this issue as related to #60905 (moved)
marked this issue as related to #60907 (moved)
@kencjohnston - I apologize for the very late response on this, but if you are still looking to have this conversation, I think that @syasonik might have the best context around the source-controlled dashboards and alerts.
Hey!
GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current
gitlab-cerepository will become a read-only mirror, without any proprietary code. All development is moved to the currentgitlab-eerepository, which we will rename to justgitlabin the coming weeks. As part of this migration, issues will be moved to the currentgitlab-eeproject.If you have any questions about all of this, please ask them in our dedicated FAQ issue.
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855You can also find more information here:
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304- https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/
Frequently Asked Questions
For an up to date list of questions and answers, please take a look at
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855What will we do with the repository names?
https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.
Why rename gitlab-ce to gitlab-foss?
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear
I created a merge requests for CE, and this got closed. What do I need to do?
You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.
How does the licensing work in this new setup?
Everything in the
ee/directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in theee/directory, the process of contributing changes is the same as when using the gitlab-ce repository.Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?
Yes, these changes will be synced to gitlab-foss automatically, several times per day.
