The purpose of this spike is to try and create a custom o11y(metrics) dashboard by utilising the custom dashboards framework, and highlight any pain points, issues or improvements ideas, with the overall goal of making the framework easier to use, and not tied to PA.
I think this already kind of highlights the blurred boundaries between what's a purely UI framework and the analytics dashboard framework itself.
By looking at the code, the Customisable Dashboard on its own is currently more of a UI framework, and as such, it should probably be moved to gitlab-ui ( there is already some existing issues for this gitlab-ui#2764 (comment 1979655594) which @jiaan created), and the UX of panels, dashboard, filters etc be well defined and scoped there (cc @lvanc ).
Datasources handling on the other hand are now currently handled by the Analytics Dashboard framework .
Question: What do we want this framework to be? Do we want to make the analytics dashboard framework part of the Customisable Dashboard framework? Or keep the custom dashboard framework purely a UI framework, and make the analytics dashboard a reusable opinionated framework, not tied to product analytics, and usable by any team from any stage that wants to create a dashboard that is hooked up to a datasource?
I've finished playing around with dashboards finally .
I've managed to successfully create a dashboard for o11y metrics, with all the supported methods: 1. programmatically, 2. yaml config 3. dashboard + viz designer.
Here are the notes that I took along the way, which some proposed action items:
When going through the customizable_dashboards docs, there is a bit of confusion around data sources. The docs guide you through creating a datasource, but the customizable_dashboards framework does not actually support them on its own, being mostly just a set of UI components. It's the analytics-dashboard framework that handles datasource (see AnalyticsPanel )
Action item: Review the definition and boundaries between customisable dashboard framework and analytics dashboard framework. More specifically:
Customisable Dashboard framework could be purely a UI library of components. As such, after cleaning that up from unnecessary business logic, we should move most/all of it to gitlab-ui ( some effort was already tracked here: gitlab-ui#2764 (comment 1979655594) ), so that a consistent reusable UX can be established and document properly. Docs should be updated accordingly, and remove any reference to concepts that belong to Analytics Dashboard
Analytics Dashboard framework can instead be a more opinionated framework, which makes use of the Customisable Dashboard framework, and supports datasources and more advanced toolings. See next points for possible improvements.
When creating the dashboard programmatically, the example config used in the docs are outdated. I've found in the codebase usage of utility methods such as createNewVisualizationPanel. We should probably update the docs and suggest to use the utility methods instead. Probably worth creating one for dashboard and visualization as well, as manually editing them is a bit cumbersome. I would avoid exposing the internals of the config itself if possible.
Action item: Make the relevant config private and remove them from the docs. Instead create/reuse API to build such config, and document them properly
Currently the dashboard designer is part of the customisable dashboard framework, but it's not 100% reusable outside of PA context ( I had some errors when trying it out), while the data explorer ( former viz designer ) is currently part of the analytics dashboard framework and tied to PA.
Action item:
Given that the data explorer is a tool to create visualisations from a datasource, I think it makes sense to keep it as part of the analytics dashboard framework. We just need to make it less PA specific (#500475) and more easy to add datasources etc ( some of this was discussed as part of &15655 )
Customisable Dashboard has embedded support for date filters and anonymous users. Maybe we could improve the API so that filters can be defined by the user of the dashboard, rather than having to rely on the default ones.
Action item:
Remove filters from Customisable Dashboards and add/improve API to define filters externally. We can then move current filters handling to the analytics dashboard implementation. This should ideally give us the chance to improve filters handling in general, and being able to have filters specific to datasources, which is something we discussed already.
Being date filters probably quite a popular one, we should make it so that it's easier to create a date filter and override change its options. I've also found an quirk where when set to Today, the end date and start date is set to the same data ( i.e. today ), and that does not work for o11y ( delta = 0 ). So we might want to take a look at some of those quirks as well.
Creating a datasource is not much documented at the moment. And I think the process is general could be streamlined, especially if we plan to open it up to more sources.
Action item: Improve support for external datasources across the whole framework and document it ( both in the docs and code documentation )
Using the dashboard designer with data explorer is very nice. But it would be even nicer to be able to create viz from external charts rather than having to recreate them in the editor. This is something we discussed already about adding a quick "Add to dashboard" button that we could add to existing charts.
Action item: Add support to create viz from external existing charts.
There are some ad-hoc fixes in the customisable dashboard code that needs to be removed to make the framework agnostic from where its used (#465569, 93e0ce0d)
There are some apollo related utils (i.e. updateApolloCache ) inside Customisable Dashboard framework that are used by analytics_dashboard. We should move them out
Very minor, but when creating YAML files only .yaml is recognised. .yml is not. Is that intended? If not, can be a quick improvements. Also it seems like the viz name and filenames must match, which I did not see mentioned in the docs (unless I'm mistaken)
o11y data does not play well with the existing charts, i.e. legends and tooltip do not render correctly. We should look whether this might affect others ( Optimise? )
Do we want users to be able to use the Analytics Dashboard framework and create dashboards anywhere, or can analytics dashboard only exist under the analytics_dashboards/ slice of the codebase?
Similarly, can we think of other use cases where users might want to create dashboards that are not analytics dashboard, but still need to support fetching data from a datasource? If that's the case, then maybe the customizable dashboard framework does not need to be responsible only for the UI, but we can also move data source handling from analytics to the framework.
While reviewing an unrelated MRs, I've learned that to add a viz type one also need to add it to the schema validator as well (see ee/lib/gitlab/analytics/value_stream_dashboard/visualizations + ee/app/models/product_analytics/visualization.rb). Is that actually needed? I could not see it documented anywhere?
Thank you for running through this @drosse! This is a great summary, and I agree with the overall themes . I think if we can move the framework UI into GitLab UI, and make the analytics dashboards the new shared component, it would clear up a lot of the confusion. I think we can start making issues based off this list, adding them to Decouple Product Analytics from Dashboard frame... (&15691). WDYT?
Being date filters probably quite a popular one, we should make it so that it's easier to create a date filter and override change its options. I've also found an quirk where when set to Today, the end date and start date is set to the same data ( i.e. today ), and that does not work for o11y ( delta = 0 ). So we might want to take a look at some of those quirks as well.
Could you expand on this? When you select Today, what values do you expect the start and end dates to be? Is it more the time of day, rather than the dates themselves?
Very minor, but when creating YAML files only .yaml is recognised. .yml is not. Is that intended? If not, can be a quick improvements. Also it seems like the viz name and filenames must match, which I did not see mentioned in the docs (unless I'm mistaken)
Yep, support of .yaml and .yml is a bit mismatched on the backend and frontend, as well as which extension is the default one used too. We should definitely make sure both are supported across the entire stack.
o11y data does not play well with the existing charts, i.e. legends and tooltip do not render correctly. We should look whether this might affect others ( Optimise? )
O11y legends and tooltips are what I would class as a more “advanced” use-case. Since O11y needs to support trace linking and complex data points. Having said that, I agree that we should make sure to support this kind of use-case, even if the default is more basic.
Adding a new viz render type requires importing the component directly into the panel component. Can we find a less intrusive way to do this?
Possibly. The issue is importing into Vue without importing every potential visualization type on load . Right now it's configuration-based, but we could look at basing it on convention instead. For instance, put your visualization file in x folder, and we'll automatically import it when used.
Do we want users to be able to use the Analytics Dashboard framework and create dashboards anywhere, or can analytics dashboard only exist under the analytics_dashboards/ slice of the codebase?
I would suggest this is more of a product decision for each team. We have a UI structure that works for analytics:
Dashboard listing
Dashboard
Data explorer
Dashboard editor
But another area of the product might only need 1 or a few of these aspects. For instance, a static dashboards page using the framework UI. I can definitely see a scenario where we move the analytics dashboards system over to a shared component structure, to give product areas a bit of a leg up if they want the more advanced features. We just need to continue separating our code from the shared component.
Similarly, can we think of other use cases where users might want to create dashboards that are not analytics dashboard, but still need to support fetching data from a datasource? If that's the case, then maybe the customizable dashboard framework does not need to be responsible only for the UI, but we can also move data source handling from analytics to the framework.
Since the framework is moving to GitLab UI, I think it should be data agnostic. I think if a team wants to build a dashboard without using the analytics dashboard structure, then it should be up to them how they leverage that. It could be an API request, HAML embed, or even moving the data source logic to a shared component and importing that.
While reviewing an unrelated MRs, I've learned that to add a viz type one also need to add it to the schema validator as well (see ee/lib/gitlab/analytics/value_stream_dashboard/visualizations + ee/app/models/product_analytics/visualization.rb). Is that actually needed? I could not see it documented anywhere?
Yeah, it is needed for built-in visualization types . Its needed if you want to use the customizable dashboards GraphQL endpoint, otherwise it won't pull it in. Of course, if someone isn't using the GraphQL endpoint, then they can ignore this step. I guess this falls into the analytics dashboard area rather than customizable dashboards .
While reviewing an unrelated MRs, I've learned that to add a viz type one also need to add it to the schema validator as well (see ee/lib/gitlab/analytics/value_stream_dashboard/visualizations + ee/app/models/product_analytics/visualization.rb). Is that actually needed? I could not see it documented anywhere?
Yeah, it is needed for built-in visualization types . Its needed if you want to use the customizable dashboards GraphQL endpoint, otherwise it won't pull it in. Of course, if someone isn't using the GraphQL endpoint, then they can ignore this step. I guess this falls into the analytics dashboard area rather than customizable dashboards .
Could you expand on this? When you select Today, what values do you expect the start and end dates to be? Is it more the time of day, rather than the dates themselves?
yes, o11y API are time sensible, so getting data for "today" means having a date/time span from 00:00 to 23:59.
Issues created, linked to the epic and also referenced in the summary above. I just moved my notes in each issue description to start, we can then refine and better scope them out.
I think for now, we should stay focused on two things:
Getting things as tidy as possible to make it easy for existing "basic" static dashboards to be migrated to the framework. Basically, if we could move as many under the analyze heading as possible without a lot of effort, that's a huge win
Closing the gaps where Optimize had to deviate to make their VSA and AI Impact analysis dashboards
I think in that process, getting as much documented in the design system and implemented in GitLab UI as possible is rad.
For PA or o11y datasources, we will need to essentially hide those for customers, as those features now will both be internal only. We can use them in our discussions on how we onboard new datasources, but they shouldn't be visible to customers for now.
Getting things as tidy as possible to make it easy for existing "basic" static dashboards to be migrated to the framework. Basically, if we could move as many under the analyze heading as possible without a lot of effort, that's a huge win
All those tasks I've created should help moving towards this. Some might be "generic" and not impact Optimise directly, but, soon or later, we should still address them
Closing the gaps where Optimize had to deviate to make their VSA and AI Impact analysis dashboards
Some good points were made here, but I guess there might be more we need to work on. We need to discuss this with Optimise asap and make a plan.
I think in that process, getting as much documented in the design system and implemented in GitLab UI as possible is rad.
For PA or o11y datasources, we will need to essentially hide those for customers, as those features now will both be internal only. We can use them in our discussions on how we onboard new datasources, but they shouldn't be visible to customers for now.
Sounds good. o11y is not integrated anyway (I only built a prototype, nothing was merged into prod).
And just to make to I have a full picture, the dashboards that Optimise owns are targeted for GitLab external users as so called "built by gitlab" dashboards right? They are not for GitLab internal use only are they?
Correct, all of the optimize dashboards are live and for external customers. The "built by GitLab" label is just our way of saying that we built a prescriptive starter dashboard for you to analyze this data source.
And external customers should be able to create more dashboards, querying the same Optimise datasource I presume? Should the data explorer be available to external users as well? Or are external users limited to the predefined vizs we created?
That's what I was sort of asking Alex in the follow up comment.
Step one: they simply get their existing prebuilt dashboards fully ported to the framework and the user can view those and only make simple modifications like time changes or layout edits. This means we have those dashboards needs all documented in Pajamas and GitLab UI. That is our collective first goal. That use case informs how quickly we can do that same migration for all the other existing static dashboards under the Analyze header (CI/CD analytics, repository analytics, merge request analytics, etc.) If we are able to migrate all of those to the frame work without needing those team's input (just straight one for one,) that'd be super cool
Step two (Advanced): custom dashboard creation, using the viz designer/data explorer, external customers can write their own queries, ask their own questions and build their own dashboards using that same Optimize data. So a user could have some charts from their Value stream analytics dashboards, and some from their AI Impact analysis and some that are specific to their own needs of that data. This is the ulitmate goal of all of this project, that all data sources in GitLab are available here to be queried by external users so they can build the dashboards they want.
That step two is what Alex mentioned isn't their top priority but on their radar.
Step one: they simply get their existing prebuilt dashboards fully ported to the framework and the user can view those and only make simple modifications like time changes or layout edits. This means we have those dashboards needs all documented in Pajamas and GitLab UI. That is our collective first goal.
Ok, that makes sense. For this I think we need to understand clearly from them what's stopping them from fully porting existing dashboards ( assuming we are talking here about the value stream and ai impact ). Some was covered by #503809 (comment 2218698243) I assume.
Also are we doing the porting or are they?
This means we have those dashboards needs all documented in Pajamas and GitLab UI.
I've created an &15999 for this work. I also think this could happen in parallel ideally and it's not necessarily a blocker to start the porting.
Daniele Rossettichanged title from Spike - Create a custom o11y/metrics dashboard with dashboard framework to Spike - Create a custom o11y-metrics dashboard with dashboard framework
changed title from Spike - Create a custom o11y/metrics dashboard with dashboard framework to Spike - Create a custom o11y-metrics dashboard with dashboard framework
Daniele Rossettichanged the descriptionCompare with previous version