Update analytics cube proxy to use POST (!103330) · Merge requests · GitLab.org / GitLab

Robert Hunt requested to merge 381696-add-support-for-post-requests-in-our-use-of-cube-s-js-library into master Nov 08, 2022

What does this MR do and why?

In Update Cube proxy to work with Cube JS library (!100714 - merged) we changed the cube proxy to use GET because the Cube JS library uses it by default and trying to use POST resulted in failures because the headers weren't being shared correctly.

Since then, the concern that the query character limit of 2048 may be reached when running Cube queries (and the fact the library forces POST on queries over 2000 characters), has meant that I've had another go at making it work.

By reviewing how axios makes POST requests, it was determined that we can in fact pass the necessary headers for authentication purposes, and in doing so, it means that we can make this endpoint a POST request once again.

frontend implementation

import { CubejsApi, HttpTransport } from '@cubejs-client/core';
import csrf from '~/lib/utils/csrf';

const cubejsApi = new CubejsApi('1', {
  transport: new HttpTransport({
    apiUrl: '/api/v4/projects/1/product_analytics/request',
    method: 'POST',
    headers: {
      [csrf.headerKey]: csrf.token,
      'X-Requested-With': 'XMLHttpRequest',
    },
    credentials: 'same-origin',
  }),
});

This API endpoint is behind a default-off feature flag and is not in use by our own code just yet, so it swapping it now is the easiest thing to do.

Screenshots or screen recordings

N/A

How to set up and validate locally

Make sure you are on at least GitLab Premium.

Testing this MR specifically

Enable the feature flag: echo "Feature.enable(:cube_api_proxy)" | rails c.
Set up the devkit and start it running using docker-compose up.
Update application settings: echo "ApplicationSetting.current.update(cube_api_base_url: 'http://gdk.test:4000', cube_api_key: API KEY FROM CUBE .env)" | rails c.
As an authenticated user who has developer+ access to a particular project, make the following API call:

POST /api/v4/projects/{PROJECT_ID}/product_analytics/request/load

Content-Type: application/json

Send it with a valid cube query as a JSON body:

{
  "query": {
    "measures": [
      "Jitsu.count"
    ],
    "timeDimensions": [
      {
        "dimension": "Jitsu.utcTime",
        "dateRange": "This week"
      }
    ],
    "order": [
      [
        "Jitsu.count",
        "desc"
      ],
      [
        "Jitsu.docPath",
        "desc"
      ],
      [
        "Jitsu.utcTime",
        "asc"
      ]
    ],
    "dimensions": [
      "Jitsu.docPath"
    ],
    "limit": 23
  },
  "queryType": "multi"
}

Expect it to respond with a 201

Testing with a Cube JS graph (for how it will be used in future, only test if you **really** want to)

Set up the devkit and start it running using docker-compose up.
Open the rails console: rails c
Enable the feature flag: Feature.enable(:cube_api_proxy).
Update the application settings (details obtained through devkit setup):

ApplicationSetting.current.update(jitsu_project_xid: 'XXXXXXXXXX', jitsu_host: 'http://gdk.test:8000', clickhouse_connection_string: 'http://test:test@clickhouse:8123', jitsu_administrator_email: 'XXXXXXX', jitsu_administrator_password: 'XXXXXXX', cube_api_base_url: 'http://gdk.test:4000', cube_api_key: API KEY FROM CUBE .env)
reload!

Find a project on the console: project = Project.find(PROJECT_ID)
Run ProductAnalytics::InitializeStackService.new(container: project).execute
Ensure that a sidekiq job was enqueued and in the Jitsu configurator the new API key is created.
Copy the JS API key and add the JS SDK to a local page to start generating data.
Apply the following patch to add Cube to the frontend:

Index: config/webpack.config.js
<+>UTF-8
===================================================================
diff --git a/config/webpack.config.js b/config/webpack.config.js
--- a/config/webpack.config.js	(revision Staged)
+++ b/config/webpack.config.js	(date 1667833878228)
@@ -306,6 +306,11 @@
         test: /\.mjs$/,
         use: [],
       },
+      {
+        test: /(@cubejs-client\/vue).*\.(js)?$/,
+        include: /node_modules/,
+        loader: 'babel-loader',
+      },
       WEBPACK_USE_ESBUILD_LOADER && {
         test: /\.js$/,
         exclude: (modulePath) =>
Index: ee/app/assets/javascripts/product_analytics/dashboards/components/widgets/cube_line_chart.vue
<+>UTF-8
===================================================================
diff --git a/ee/app/assets/javascripts/product_analytics/dashboards/components/widgets/cube_line_chart.vue b/ee/app/assets/javascripts/product_analytics/dashboards/components/widgets/cube_line_chart.vue
--- a/ee/app/assets/javascripts/product_analytics/dashboards/components/widgets/cube_line_chart.vue	(revision Staged)
+++ b/ee/app/assets/javascripts/product_analytics/dashboards/components/widgets/cube_line_chart.vue	(date 1667841400276)
@@ -1,8 +1,40 @@
 <script>
-import { s__ } from '~/locale';
+import { QueryRenderer } from '@cubejs-client/vue';
+import { GlLineChart } from '@gitlab/ui/dist/charts';
+import { CubejsApi, HttpTransport } from '@cubejs-client/core';
+import csrf from '~/lib/utils/csrf';
+
+const cubejsApi = new CubejsApi('1', {
+  transport: new HttpTransport({
+    apiUrl: '/api/v4/projects/1/product_analytics/request',
+    method: 'POST',
+    headers: {
+      [csrf.headerKey]: csrf.token,
+      'X-Requested-With': 'XMLHttpRequest',
+    },
+    credentials: 'same-origin',
+  }),
+});
+
+const overTimeQueries = {
+  users: {
+    measures: ['Jitsu.count'],
+    timeDimensions: [
+      {
+        granularity: 'day',
+        dimension: 'Jitsu.utcTime',
+      },
+    ],
+    dimensions: ['Jitsu.eventType'],
+  },
+};
 
 export default {
   name: 'CubeLineChart',
+  components: {
+    GlLineChart,
+    QueryRenderer,
+  },
   props: {
     data: {
       type: Object,
@@ -20,12 +52,57 @@
       default: () => ({}),
     },
   },
-  i18n: {
-    content: s__('ProductAnalytics|Widgets content'),
+  data() {
+    return { cubejsApi, query: this.withFilters(overTimeQueries.users, 'day') };
+  },
+  methods: {
+    series(resultSet) {
+      if (!resultSet) {
+        return [];
+      }
+
+      const seriesNames = resultSet?.seriesNames();
+      const pivot = resultSet?.chartPivot();
+      const series = [];
+
+      seriesNames.forEach((e) => {
+        const data = pivot.map((p) => [p.x, p[e.key]]);
+        series.push({
+          name: e.title,
+          data,
+        });
+      });
+
+      return series;
+    },
+    withFilters(query, granularity) {
+      const newQuery = {
+        ...query,
+      };
+
+      newQuery.timeDimensions = [
+        {
+          dimension: `Jitsu.utcTime`,
+          dateRange: [new Date(2022, 8), new Date(2022, 10)],
+          granularity: granularity || null,
+        },
+      ];
+
+      return newQuery;
+    },
   },
 };
 </script>
 
 <template>
-  <p>{{ $options.i18n.content }}</p>
+  <query-renderer :cubejs-api="cubejsApi" :query="query">
+    <template #default="{ resultSet }">
+      <gl-line-chart
+        :data="series(resultSet)"
+        :option="chartOptions"
+        :show-legend="!customizations.hideLegend"
+        responsive
+      />
+    </template>
+  </query-renderer>
 </template>
Index: jest.config.base.js
<+>UTF-8
===================================================================
diff --git a/jest.config.base.js b/jest.config.base.js
--- a/jest.config.base.js	(revision Staged)
+++ b/jest.config.base.js	(date 1667833913621)
@@ -153,6 +153,7 @@
     'dateformat',
     'lowlight',
     'vscode-languageserver-types',
+    '@cubejs-client/vue',
     ...gfmParserDependencies,
   ];
 
Index: package.json
<+>UTF-8
===================================================================
diff --git a/package.json b/package.json
--- a/package.json	(revision Staged)
+++ b/package.json	(date 1667833913631)
@@ -51,6 +51,8 @@
     "@babel/core": "^7.18.5",
     "@babel/preset-env": "^7.18.2",
     "@codesandbox/sandpack-client": "^1.2.2",
+    "@cubejs-client/core": "^0.31.0",
+    "@cubejs-client/vue": "^0.31.0",
     "@gitlab/at.js": "1.5.7",
     "@gitlab/favicon-overlay": "2.0.0",
     "@gitlab/svgs": "3.7.0",
Index: yarn.lock
<+>UTF-8
===================================================================
diff --git a/yarn.lock b/yarn.lock
--- a/yarn.lock	(revision Staged)
+++ b/yarn.lock	(date 1667834186006)
@@ -1041,6 +1041,27 @@
   resolved "https://registry.yarnpkg.com/@csstools/selector-specificity/-/selector-specificity-2.0.1.tgz#b6b8d81780b9a9f6459f4bfe9226ac6aefaefe87"
   integrity sha512-aG20vknL4/YjQF9BSV7ts4EWm/yrjagAN7OWBNmlbEOUiu0llj4OGrFoOKK3g2vey4/p2omKCoHrWtPxSwV3HA==
 
+"@cubejs-client/core@^0.31.0", "@cubejs-client/core@^0.31.9":
+  version "0.31.9"
+  resolved "https://registry.yarnpkg.com/@cubejs-client/core/-/core-0.31.9.tgz#5225a322a90f2dbdd3342e6b3b3ecc7abbc373ef"
+  integrity sha512-AXrAVdAcMN+WLyhHtXYMRQ+YfynUKO5zUEUORdWWPjqXf8zJl1wltLWQHlIAFs43bwjHopOTncf6sI4o7tTNAQ==
+  dependencies:
+    core-js "^3.6.5"
+    cross-fetch "^3.0.2"
+    dayjs "^1.10.4"
+    ramda "^0.27.0"
+    url-search-params-polyfill "^7.0.0"
+    uuid "^8.3.2"
+
+"@cubejs-client/vue@^0.31.0":
+  version "0.31.9"
+  resolved "https://registry.yarnpkg.com/@cubejs-client/vue/-/vue-0.31.9.tgz#19a68534589d2063539e0c6c0b03ae262d545044"
+  integrity sha512-p0s6dfefNMoywAk6PjbtHYTO6bH1wptu1c/wkjX7yl0r0KeJDHFvJX4mtWHbUmysxERgEfrgYzw0e3J2Nt8KeQ==
+  dependencies:
+    "@cubejs-client/core" "^0.31.9"
+    core-js "^3.6.5"
+    ramda "^0.27.0"
+
 "@discoveryjs/json-ext@^0.5.0":
   version "0.5.6"
   resolved "https://registry.yarnpkg.com/@discoveryjs/json-ext/-/json-ext-0.5.6.tgz#d5e0706cf8c6acd8c6032f8d54070af261bbbb2f"
@@ -3903,7 +3924,7 @@
   resolved "https://registry.yarnpkg.com/core-js-pure/-/core-js-pure-3.6.5.tgz#c79e75f5e38dbc85a662d91eea52b8256d53b813"
   integrity sha512-lacdXOimsiD0QyNf9BC/mxivNJ/ybBGJXQFKzRekp1WTHoVUWsUHEn+2T8GJAzzIhyOuXA+gOxCVN3l+5PLPUA==
 
-core-js@^3.26.0:
+core-js@^3.26.0, core-js@^3.6.5:
   version "3.26.0"
   resolved "https://registry.yarnpkg.com/core-js/-/core-js-3.26.0.tgz#a516db0ed0811be10eac5d94f3b8463d03faccfe"
   integrity sha512-+DkDrhoR4Y0PxDz6rurahuB+I45OsEUv8E1maPTB6OuHRohMMcznBq9TMpdpDMm/hUPob/mJJS3PqgbHpMTQgw==
@@ -3988,6 +4009,13 @@
   dependencies:
     jquery ">= 1.9.1"
 
+cross-fetch@^3.0.2:
+  version "3.1.5"
+  resolved "https://registry.yarnpkg.com/cross-fetch/-/cross-fetch-3.1.5.tgz#e1389f44d9e7ba767907f7af8454787952ab534f"
+  integrity sha512-lvb1SBsI0Z7GDwmuid+mU3kWVBwTVUbe7S0H52yaaAdQOXq2YktTCZdlAcNKFzE6QtRz0snpw9bNiPeOIkkQvw==
+  dependencies:
+    node-fetch "2.6.7"
+
 cross-spawn@^6.0.5:
   version "6.0.5"
   resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-6.0.5.tgz#4a5ec7c64dfae22c3a14124dbacdee846d80cbc4"
@@ -4686,6 +4714,11 @@
   resolved "https://registry.yarnpkg.com/dateformat/-/dateformat-5.0.1.tgz#60a27a2deb339f888ba4532f533e25ac73ca3d19"
   integrity sha512-DrcKxOW2am3mtqoJwBTK3OlWcF0QSk1p8diEWwpu3Mf//VdURD7XVaeOV738JvcaBiFfm9o2fisoMhiJH0aYxg==
 
+dayjs@^1.10.4:
+  version "1.11.6"
+  resolved "https://registry.yarnpkg.com/dayjs/-/dayjs-1.11.6.tgz#2e79a226314ec3ec904e3ee1dd5a4f5e5b1c7afb"
+  integrity sha512-zZbY5giJAinCG+7AGaw0wIhNZ6J8AhWuSXKvuc1KAyMiRsvGQWqh4L+MomvhdAYjN+lqvVCMq1I41e3YHvXkyQ==
+
 de-indent@^1.0.2:
   version "1.0.2"
   resolved "https://registry.yarnpkg.com/de-indent/-/de-indent-1.0.2.tgz#b2038e846dc33baa5796128d0804b455b8c1e21d"
@@ -9259,7 +9292,7 @@
   resolved "https://registry.yarnpkg.com/node-domexception/-/node-domexception-1.0.0.tgz#6888db46a1f71c0b76b3f7555016b63fe64766e5"
   integrity sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==
 
-node-fetch@^2.6.1, node-fetch@^2.6.7:
+node-fetch@2.6.7, node-fetch@^2.6.1, node-fetch@^2.6.7:
   version "2.6.7"
   resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-2.6.7.tgz#24de9fba827e3b4ae44dc8b20256a379160052ad"
   integrity sha512-ZjMPFEfVx5j+y2yF35Kzx5sF7kDzxuDj6ziH4FFbOp87zKDZNx8yExJIb05OGF4Nlt9IHFIMBkRl41VdvcNdbQ==
@@ -10292,6 +10325,11 @@
   resolved "https://registry.yarnpkg.com/quick-lru/-/quick-lru-4.0.1.tgz#5b8878f113a58217848c6482026c73e1ba57727f"
   integrity sha512-ARhCpm70fzdcvNQfPoy49IaanKkTlRWF2JMzqhcJbhSFRZv7nPTvZJdcY7301IPmvW+/p0RgIWnQDLJxifsQ7g==
 
+ramda@^0.27.0:
+  version "0.27.2"
+  resolved "https://registry.yarnpkg.com/ramda/-/ramda-0.27.2.tgz#84463226f7f36dc33592f6f4ed6374c48306c3f1"
+  integrity sha512-SbiLPU40JuJniHexQSAgad32hfwd+DRUdwF2PlVuI5RZD0/vahUco7R8vD86J/tcEKKF9vZrUVwgtmGCqlCKyA==
+
 randombytes@^2.0.0, randombytes@^2.0.1, randombytes@^2.0.5, randombytes@^2.1.0:
   version "2.1.0"
   resolved "https://registry.yarnpkg.com/randombytes/-/randombytes-2.1.0.tgz#df6f84372f0270dc65cdf6291349ab7a473d4f2a"
@@ -12119,6 +12157,11 @@
     mime-types "^2.1.27"
     schema-utils "^3.0.0"
 
+url-search-params-polyfill@^7.0.0:
+  version "7.0.1"
+  resolved "https://registry.yarnpkg.com/url-search-params-polyfill/-/url-search-params-polyfill-7.0.1.tgz#b900cd9a0d9d2ff757d500135256f2344879cbff"
+  integrity sha512-bAw7L2E+jn9XHG5P9zrPnHdO0yJub4U+yXJOdpcpkr7OBd9T8oll4lUos0iSGRcDvfZoLUKfx9a6aNmIhJ4+mQ==
+
 url@^0.11.0:
   version "0.11.0"
   resolved "https://registry.yarnpkg.com/url/-/url-0.11.0.tgz#3838e97cfc60521eb73c525a8e55bfdd9e2e28f1"

Run yarn install on the root of your GitLab instance and restart your GDK
Visit any dashboards such as http://gdk.test:3000/gitlab-org/gitlab-test/-/product_analytics/dashboards/dashboard_audience and validate that the widget appears with a chart being shown and the API request returns 201.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Related to #381696 (closed)

Edited Nov 08, 2022 by Robert Hunt

Update analytics cube proxy to use POST

What does this MR do and why?

Screenshots or screen recordings

How to set up and validate locally

MR acceptance checklist

Merge request reports