Merge branch '444915-improve-schema-errors-with-custom-validation' into 'master'

Improve Analytics dashboards schema errors

See merge request !153152



Merged-by: Felipe Artur <fcardozo@gitlab.com>
Approved-by: Tyler Amos <tamos@gitlab.com>
Approved-by: Alex Pennells <apennells@gitlab.com>
Approved-by: Felipe Artur <fcardozo@gitlab.com>
Approved-by: Elwyn Benson <ebenson@gitlab.com>
Approved-by: Libor Vanc <5022246-lvanc@users.noreply.gitlab.com>
Approved-by: Lorena Ciutacu <lciutacu@gitlab.com>
Reviewed-by: Felipe Artur <fcardozo@gitlab.com>
Reviewed-by: Alex Pennells <apennells@gitlab.com>
Reviewed-by: Elwyn Benson <ebenson@gitlab.com>
Reviewed-by: Rudy Crespo <rcrespo@gitlab.com>
Co-authored-by: Rudy Crespo <rcrespo@gitlab.com>

doc/api/graphql/reference/index.md

@@ -18979,10 +18979,10 @@ Represents a product analytics dashboard.
 | <a id="customizabledashboardconfigurationproject"></a>`configurationProject` | [`Project`](#project) | Project which contains the dashboard definition. |
 | <a id="customizabledashboarddescription"></a>`description` | [`String`](#string) | Description of the dashboard. |
 | <a id="customizabledashboarderrors"></a>`errors` | [`[String!]`](#string) | Errors on yaml definition. |
-| <a id="customizabledashboardpanels"></a>`panels` | [`CustomizableDashboardPanelConnection!`](#customizabledashboardpanelconnection) | Panels shown on the dashboard. (see [Connections](#connections)) |
+| <a id="customizabledashboardpanels"></a>`panels` | [`CustomizableDashboardPanelConnection`](#customizabledashboardpanelconnection) | Panels shown on the dashboard. (see [Connections](#connections)) |
 | <a id="customizabledashboardslug"></a>`slug` | [`String!`](#string) | Slug of the dashboard. |
 | <a id="customizabledashboardstatus"></a>`status` **{warning-solid}** | [`String`](#string) | **Introduced** in GitLab 17.0. **Status**: Experiment. Status of the dashboard. |
-| <a id="customizabledashboardtitle"></a>`title` | [`String!`](#string) | Title of the dashboard. |
+| <a id="customizabledashboardtitle"></a>`title` | [`String`](#string) | Title of the dashboard. |
 | <a id="customizabledashboarduserdefined"></a>`userDefined` | [`Boolean!`](#boolean) | Indicates whether the dashboard is user-defined or provided by GitLab. |
 ### `CustomizableDashboardPanel`
@@ -18995,8 +18995,8 @@ Represents a product analytics dashboard panel.
 | ---- | ---- | ----------- |
 | <a id="customizabledashboardpanelgridattributes"></a>`gridAttributes` | [`JSON`](#json) | Description of the position and size of the panel. |
 | <a id="customizabledashboardpanelqueryoverrides"></a>`queryOverrides` | [`JSON`](#json) | Overrides for the visualization query object. |
-| <a id="customizabledashboardpaneltitle"></a>`title` | [`String!`](#string) | Title of the panel. |
-| <a id="customizabledashboardpanelvisualization"></a>`visualization` | [`CustomizableDashboardVisualization!`](#customizabledashboardvisualization) | Visualization of the panel. |
+| <a id="customizabledashboardpaneltitle"></a>`title` | [`String`](#string) | Title of the panel. |
+| <a id="customizabledashboardpanelvisualization"></a>`visualization` | [`CustomizableDashboardVisualization`](#customizabledashboardvisualization) | Visualization of the panel. |
 ### `CustomizableDashboardVisualization`

doc/user/analytics/analytics_dashboards.md

@@ -313,6 +313,10 @@ If the error persists:
 - Check that your configurations match the [dashboard JSON schema](#define-a-dashboard) defined in `ee/app/validators/json_schemas/analytics_dashboard.json`.
 - For product analytics, make sure your [admin and project settings](../product_analytics/index.md#project-level-settings) are set up correctly.
 
+### `Invalid dashboard configuration`
+
+If the dashboard displays a global error message that the configuration is invalid, check that your configurations match the [dashboard JSON schema](#define-a-dashboard) defined in `ee/app/validators/json_schemas/analytics_dashboard.json`.
+
 ### `Invalid visualization configuration`
 
 If a dashboard panel displays a message that the visualization configuration is invalid,

doc/user/analytics/value_streams_dashboard.md

@@ -276,10 +276,13 @@ After you have set up the project, set up the configuration file:
 | `height` (subfield of `gridAttributes`) | Height of the panel |
 
 ```yaml
-# title - Change the title of the Value Streams Dashboard. [optional]
+# version - The latest version of the analytics dashboard schema
+version: '2'
+
+# title - Change the title of the Value Streams Dashboard.
 title: 'Custom Dashboard title'
 
-# description - Change the description of the Value Streams Dashboard. [optional]
+# description - Change the description of the Value Streams Dashboard.
 description: 'Custom description'
 
 # panels - List of panels that contain panel settings.

ee/app/assets/javascripts/analytics/analytics_dashboards/components/analytics_dashboard.vue

 <script>
-import { GlEmptyState, GlSkeletonLoader } from '@gitlab/ui';
+import { GlEmptyState, GlSkeletonLoader, GlAlert } from '@gitlab/ui';
 import * as Sentry from '~/sentry/sentry_browser_wrapper';
 import { createAlert } from '~/alert';
 import { HTTP_STATUS_BAD_REQUEST, HTTP_STATUS_CREATED } from '~/lib/utils/http_status';
@@ -55,6 +55,7 @@ export default {
     ValueStreamFeedbackBanner,
     GlEmptyState,
     GlSkeletonLoader,
+    GlAlert,
   },
   mixins: [InternalEvents.mixin(), glFeatureFlagsMixin()],
   inject: {
@@ -113,7 +114,7 @@ export default {
       backUrl: this.$router.resolve('/').href,
       changesSaved: false,
       alert: null,
-      hasDashboardError: false,
+      hasDashboardLoadError: false,
       savedPanels: null,
     };
   },
@@ -135,6 +136,12 @@ export default {
     showDashboardFilters() {
       return !HIDE_DASHBOARD_FILTERS.includes(this.currentDashboard?.slug);
     },
+    invalidDashboardErrors() {
+      return this.currentDashboard?.errors ?? [];
+    },
+    hasDashboardError() {
+      return this.hasDashboardLoadError || this.invalidDashboardErrors.length > 0;
+    },
   },
   watch: {
     initialDashboard(initialDashboard) {
@@ -202,12 +209,10 @@ export default {
           title: s__('Analytics|Failed to load dashboard'),
           message,
           messageLinks: {
-            link: helpPagePath('user/analytics/analytics_dashboards', {
-              anchor: '#troubleshooting',
-            }),
+            link: this.$options.troubleshootingUrl,
           },
         });
-        this.hasDashboardError = true;
+        this.hasDashboardLoadError = true;
       },
     },
     availableVisualizations: {
@@ -344,12 +349,31 @@ export default {
       return `panel-${slug.replaceAll('_', '-')}`;
     },
   },
+  troubleshootingUrl: helpPagePath('user/analytics/analytics_dashboards', {
+    anchor: '#troubleshooting',
+  }),
 };
 </script>
 
 <template>
   <div>
     <template v-if="currentDashboard">
+      <gl-alert
+        v-if="invalidDashboardErrors.length > 0"
+        data-testid="analytics-dashboard-invalid-config-errors"
+        class="gl-mt-4"
+        :title="s__('Analytics|Invalid dashboard configuration')"
+        :primary-button-text="__('Learn more')"
+        :primary-button-link="$options.troubleshootingUrl"
+        :dismissible="false"
+        variant="danger"
+      >
+        <ul class="gl-m-0">
+          <li v-for="errorMessage in invalidDashboardErrors" :key="errorMessage">
+            {{ errorMessage }}
+          </li>
+        </ul>
+      </gl-alert>
       <value-stream-feedback-banner v-if="showValueStreamFeedbackBanner" />
       <product-analytics-feedback-banner v-if="showProductAnalyticsFeedbackBanner" />
       <customizable-dashboard

ee/app/assets/javascripts/analytics/analytics_dashboards/components/list/dashboard_list_item.vue

@@ -28,6 +28,9 @@ export default {
     showBetaBadge() {
       return this.dashboard?.status === DASHBOARD_STATUS_BETA;
     },
+    showErrorsBadge() {
+      return this.dashboard?.errors?.length > 0;
+    },
     redirectHref() {
       return joinPaths(window.location.pathname, this.dashboard.slug);
     },
@@ -86,6 +89,16 @@ export default {
           >
             {{ __('Beta') }}
           </gl-badge>
+          <gl-badge
+            v-if="showErrorsBadge"
+            data-testid="dashboard-errors-badge"
+            class="gl-ml-2"
+            size="sm"
+            icon="error"
+            variant="danger"
+          >
+            {{ __('Contains errors') }}
+          </gl-badge>
         </div>
         <gl-truncate-text
           class="gl-line-height-normal gl-text-gray-500"

ee/app/assets/javascripts/analytics/analytics_dashboards/graphql/fragments/customizable_dashboard.fragment.graphql

@@ -5,6 +5,7 @@ fragment CustomizableDashboardFragment on CustomizableDashboardConnection {
     description
     userDefined
     status
+    errors
     panels {
       nodes {
         title

ee/app/assets/javascripts/analytics/analytics_dashboards/graphql/fragments/customizable_dashboards.fragment.graphql

@@ -5,5 +5,6 @@ fragment CustomizableDashboardsFragment on CustomizableDashboardConnection {
     description
     userDefined
     status
+    errors
   }
 }

ee/app/graphql/resolvers/product_analytics/visualization_resolver.rb

@@ -6,8 +6,6 @@ class VisualizationResolver < BaseResolver
       type ::Types::ProductAnalytics::VisualizationType, null: true
 
       def resolve
-        raise Gitlab::Graphql::Errors::ResourceNotAvailable, 'Visualization does not exist' unless object.visualization
-
         object.visualization
       end
     end

ee/app/graphql/types/product_analytics/dashboard_type.rb

@@ -9,7 +9,7 @@ class DashboardType < BaseObject
 
       field :title,
         type: GraphQL::Types::String,
-        null: false,
+        null: true,
         description: 'Title of the dashboard.'
 
       field :category,
@@ -35,7 +35,7 @@ class DashboardType < BaseObject
 
       field :panels,
         type: Types::ProductAnalytics::PanelType.connection_type,
-        null: false,
+        null: true,
         description: 'Panels shown on the dashboard.'
 
       field :user_defined,

ee/app/graphql/types/product_analytics/panel_type.rb

@@ -9,7 +9,7 @@ class PanelType < BaseObject
 
       field :title,
         type: GraphQL::Types::String,
-        null: false,
+        null: true,
         description: 'Title of the panel.'
 
       field :grid_attributes,
@@ -24,7 +24,7 @@ class PanelType < BaseObject
 
       field :visualization,
         type: Types::ProductAnalytics::VisualizationType,
-        null: false,
+        null: true,
         description: 'Visualization of the panel.',
         resolver: Resolvers::ProductAnalytics::VisualizationResolver
     end

ee/app/validators/json_schemas/analytics_dashboard.json

@@ -7,7 +7,7 @@
       "additionalProperties": false,
       "properties": {
         "version": {
-          "const": "1"
+          "const": "2"
         },
         "title": {
           "type": "string"
@@ -57,7 +57,6 @@
       },
       "required": [
         "gridAttributes",
-        "queryOverrides",
         "title",
         "visualization"
       ],
@@ -119,10 +118,16 @@
       "properties": {
         "timeDimensions": {
           "$ref": "#/definitions/TimeDimensions"
+        },
+        "namespace": {
+          "type": "string"
+        },
+        "filters": {
+          "$ref": "#/definitions/Filters"
         }
       },
       "required": [
-        "timeDimensions"
+
       ],
       "title": "QueryOverrides"
     },
@@ -142,6 +147,40 @@
         "dateRange"
       ],
       "title": "TimeDimensions"
+    },
+    "Filters": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "excludeMetrics": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "include": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "labels": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "projectTopics": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        }
+      },
+      "required": [
+
+      ],
+      "title": "Filters"
     }
   }
 }

ee/lib/gitlab/analytics/ai_impact_dashboard/dashboard.yaml

 ---
 title: AI impact analytics
 description: Visualize the relation between AI usage and SDLC trends.
+version: "2"
 status: beta
 panels:
   - title: 'Metric trends for %{namespaceType}: %{namespaceName}'

ee/lib/gitlab/analytics/product_analytics/dashboards/audience.yaml

 ---
 title: Audience
 description: Understand your audience
+version: "2"
 panels:
   - visualization: daily_active_users
     title: Daily Active Users

ee/lib/gitlab/analytics/product_analytics/dashboards/behavior.yaml

 ---
 title: Behavior
 description: All insights on user activity
+version: "2"
 panels:
   - visualization: page_views_over_time
     title: Page views over time

ee/lib/gitlab/analytics/value_stream_dashboard/dashboards/value_streams_dashboard.yaml

 ---
 title: Value Streams Dashboard
 description: The Value Streams Dashboard allows all stakeholders from executives to individual contributors to identify trends, patterns, and opportunities for software development improvements.
+version: "2"
 panels:
-  - id: 1
-    visualization: dora_chart
+  - visualization: dora_chart
     title: Metrics comparison for %{namespaceName} %{namespaceType}
     gridAttributes:
       yPos: 1
@@ -11,8 +11,7 @@ panels:
       width: 12
       height: 6
     options: {}
-  - id: 2
-    visualization: usage_overview
+  - visualization: usage_overview
     title: Usage overview for %{namespaceName} %{namespaceType}
     gridAttributes:
       yPos: 0
@@ -20,8 +19,7 @@ panels:
       width: 12
       height: 1
     options: {}
-  - id: 3
-    visualization: dora_performers_score
+  - visualization: dora_performers_score
     title: DORA performers score for %{namespaceName} %{namespaceType}
     gridAttributes:
       yPos: 7

ee/spec/fixtures/product_analytics/dashboard_example_1.yaml

 ---
-version: '1'
+version: '2'
 title: Dashboard Example 1
 description: North Star Metrics across all departments for the last 3 quarters.
 panels:

ee/spec/frontend/analytics/analytics_dashboards/components/analytics_dashboard_spec.js

@@ -48,6 +48,8 @@ import {
   TEST_VISUALIZATIONS_GRAPHQL_SUCCESS_RESPONSE,
   createDashboardGraphqlSuccessResponse,
   getGraphQLDashboard,
+  TEST_INVALID_CUSTOM_DASHBOARD_GRAPHQL_SUCCESS_RESPONSE,
+  mockInvalidDashboardErrors,
 } from '../mock_data';
 
 jest.mock('~/sentry/sentry_browser_wrapper');
@@ -88,6 +90,8 @@ describe('AnalyticsDashboard', () => {
   const findProductAnalyticsFeedbackBanner = () =>
     wrapper.findComponent(ProductAnalyticsFeedbackBanner);
   const findValueStreamFeedbackBanner = () => wrapper.findComponent(ValueStreamFeedbackBanner);
+  const findInvalidDashboardAlert = () =>
+    wrapper.findByTestId('analytics-dashboard-invalid-config-errors');
 
   const mockSaveDashboardImplementation = async (responseCallback, dashboardToSave = dashboard) => {
     saveCustomDashboard.mockImplementation(responseCallback);
@@ -269,6 +273,14 @@ describe('AnalyticsDashboard', () => {
       expect(findDashboard().exists()).toBe(true);
     });
 
+    it('should not render invalid dashboard alert', async () => {
+      createWrapper();
+
+      await waitForPromises();
+
+      expect(findInvalidDashboardAlert().exists()).toBe(false);
+    });
+
     it('should add unique panel ids to each panel', async () => {
       createWrapper();
 
@@ -402,6 +414,37 @@ describe('AnalyticsDashboard', () => {
     });
   });
 
+  describe("when the dashboard's configuration is invalid", () => {
+    beforeEach(() => {
+      mockDashboardResponse(TEST_INVALID_CUSTOM_DASHBOARD_GRAPHQL_SUCCESS_RESPONSE);
+
+      createWrapper();
+
+      return waitForPromises();
+    });
+
+    it('does not render the loader', () => {
+      expect(findLoader().exists()).toBe(false);
+    });
+
+    it('renders the dashboard', () => {
+      expect(findDashboard().exists()).toBe(true);
+    });
+
+    it('renders an alert with error messages', () => {
+      expect(findInvalidDashboardAlert().props()).toMatchObject({
+        title: 'Invalid dashboard configuration',
+        primaryButtonText: 'Learn more',
+        primaryButtonLink: '/help/user/analytics/analytics_dashboards#troubleshooting',
+        dismissible: false,
+      });
+
+      mockInvalidDashboardErrors.forEach((error) =>
+        expect(findInvalidDashboardAlert().text()).toContain(error),
+      );
+    });
+  });
+
   describe('available visualizations', () => {
     it('fetches the available visualizations when a custom dashboard is loaded', async () => {
       await setupDashboard(TEST_CUSTOM_DASHBOARD_GRAPHQL_SUCCESS_RESPONSE);

ee/spec/frontend/analytics/analytics_dashboards/components/list/dashboard_list_item_spec.js

@@ -2,7 +2,10 @@ import { GlIcon, GlTruncateText } from '@gitlab/ui';
 import { visitUrl } from '~/lib/utils/url_utility';
 import DashboardListItem from 'ee/analytics/analytics_dashboards/components/list/dashboard_list_item.vue';
 import { shallowMountExtended } from 'helpers/vue_test_utils_helper';
-import { TEST_ALL_DASHBOARDS_GRAPHQL_SUCCESS_RESPONSE } from '../../mock_data';
+import {
+  mockInvalidDashboardErrors,
+  TEST_ALL_DASHBOARDS_GRAPHQL_SUCCESS_RESPONSE,
+} from '../../mock_data';
 
 jest.mock('ee/analytics/analytics_dashboards/api/dashboards_api');
 
@@ -26,6 +29,12 @@ const BETA_DASHBOARD = {
   slug: '/slug',
   status: 'beta',
 };
+const INVALID_DASHBOARD = {
+  title: 'title',
+  description: 'description',
+  slug: '/slug',
+  errors: mockInvalidDashboardErrors,
+};
 
 describe('DashboardsListItem', () => {
   /** @type {import('helpers/vue_test_utils_helper').ExtendedWrapper} */
@@ -38,6 +47,7 @@ describe('DashboardsListItem', () => {
   const findRouterLink = () => wrapper.findByTestId('dashboard-router-link');
   const findDescriptionTruncate = () => wrapper.findComponent(GlTruncateText);
   const findBetaBadge = () => wrapper.findByTestId('dashboard-beta-badge');
+  const findErrorsBadge = () => wrapper.findByTestId('dashboard-errors-badge');
 
   const $router = {
     push: jest.fn(),
@@ -86,6 +96,10 @@ describe('DashboardsListItem', () => {
       expect(findBetaBadge().exists()).toBe(false);
     });
 
+    it('does not render errors badge', () => {
+      expect(findErrorsBadge().exists()).toBe(false);
+    });
+
     it('routes to the dashboard when a list item is clicked', async () => {
       await findListItem().trigger('click');
 
@@ -128,4 +142,18 @@ describe('DashboardsListItem', () => {
       expect(findBetaBadge().exists()).toBe(true);
     });
   });
+
+  describe('with an invalid dashboard', () => {
+    beforeEach(() => {
+      createWrapper(INVALID_DASHBOARD);
+    });
+
+    it('renders the errors badge', () => {
+      expect(findErrorsBadge().props()).toMatchObject({
+        icon: 'error',
+        variant: 'danger',
+      });
+      expect(findErrorsBadge().text()).toBe('Contains errors');
+    });
+  });
 });

ee/spec/frontend/analytics/analytics_dashboards/mock_data.js

@@ -65,6 +65,7 @@ export const getGraphQLDashboard = (options = {}, withPanels = true) => {
     status: null,
     description: 'Understand your audience',
     __typename: 'CustomizableDashboard',
+    errors: [],
     ...options,
   };
 
@@ -288,6 +289,36 @@ export const TEST_ALL_DASHBOARDS_GRAPHQL_SUCCESS_RESPONSE = {
   },
 };
 
+export const mockInvalidDashboardErrors = [
+  'root is missing required keys: version',
+  "property '/panels/0' is missing required keys: queryOverrides",
+  "property '/panels/0/id' is invalid: error_type=schema",
+  "property '/panels/1' is missing required keys: queryOverrides",
+  "property '/panels/1/id' is invalid: error_type=schema",
+  "property '/panels/2' is missing required keys: queryOverrides",
+  "property '/panels/2/id' is invalid: error_type=schema",
+];
+
+export const TEST_INVALID_CUSTOM_DASHBOARD_GRAPHQL_SUCCESS_RESPONSE = {
+  data: {
+    project: {
+      id: 'gid://gitlab/Project/1',
+      customizableDashboards: {
+        nodes: [
+          getGraphQLDashboard({
+            slug: 'custom_dashboard',
+            title: 'Custom Dashboard',
+            userDefined: true,
+            errors: mockInvalidDashboardErrors,
+          }),
+        ],
+        __typename: 'CustomizableDashboardConnection',
+      },
+      __typename: 'Project',
+    },
+  },
+};
+
 export const createDashboardGraphqlSuccessResponse = (dashboardNodes) => ({
   data: {
     project: {

ee/spec/graphql/resolvers/product_analytics/visualization_resolver_spec.rb

@@ -30,8 +30,8 @@
         end
       end
 
-      it 'raises an error' do
-        expect(subject).to be_a(Gitlab::Graphql::Errors::ResourceNotAvailable)
+      it 'returns nil' do
+        expect(subject).to be_nil
       end
     end
   end