Generate a cube query using Vertex LLM
What does this MR do and why?
- Adds a new AI Action
:generate_cube_query
that takes a question and returns a query that is a valid Cube query using Cube query format.
Things this MR is missing
-
Specs -
Battle-testing the LLM prompt -
Authorization, currently anybody can use it. -
Prompt injection protection against access to other projects' data -
...probably other things
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Check out branch
- Enable AI features locally: https://docs.gitlab.com/ee/development/ai_features/#test-ai-features-locally (You might be able to skip the embedding database, I'm not sure)
- Make sure you've enabled Vertex access and configured it with your GDK. This action does not use Anthropic's API, only Vertex.
- Enable the
:expanded_ai_logging
feature flag locally. - Enable the
:generate_cube_query
flag locally. NEW - Update the code here from
perform_async
toperform_inline
. (This just makes testing easier, don't commit it anywhere.) - In another terminal, tail the LLM logs in your GDK:
tail -f log/llm.log
- Make a GraphQL call like this - feel free to update the
question
parameter:
mutation {
aiAction(input: {generateCubeQuery: {question: "How many unique users used the application this week?", resourceId: "gid://gitlab/Project/19"}, clientSubscriptionId: "ARandomID"}) {
clientMutationId
errors
}
}
- Assuming everything was set up correctly, you should (after a second or two), see a response from Vertex with a cube query
This will also be send to any open GraphQL subscriptions, but looking at the logs is a quicker way to see that it's worked.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Merge request reports
Activity
assigned to @mwoolf
- A deleted user
added backend documentation labels
- Resolved by 🤖 GitLab Bot 🤖
Proper labels assigned to this merge request. Please ignore me.
@mwoolf - please see the following guidance and update this merge request.1 Error Please add typebug typefeature, or typemaintenance label to this merge request. Edited by 🤖 GitLab Bot 🤖
mentioned in issue #435258 (closed)
added groupproduct analytics label
removed documentation label
added devopsmonitor sectionanalytics labels
added featureaddition typefeature labels
E2E Test Result Summary
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for 71077d92expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Govern | 67 | 0 | 0 | 0 | 67 | ✅ | | Create | 54 | 0 | 7 | 0 | 61 | ✅ | | Data Stores | 23 | 0 | 0 | 0 | 23 | ✅ | | Verify | 31 | 0 | 0 | 0 | 31 | ✅ | | Plan | 54 | 0 | 1 | 0 | 55 | ✅ | | Monitor | 7 | 0 | 0 | 0 | 7 | ✅ | | Package | 15 | 0 | 1 | 0 | 16 | ✅ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | | Release | 5 | 0 | 0 | 0 | 5 | ✅ | | Manage | 0 | 0 | 1 | 0 | 1 | ➖ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 258 | 0 | 10 | 0 | 268 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+
e2e-package-and-test:
test report for 71077d92expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Monitor | 24 | 0 | 8 | 0 | 32 | ✅ | | Create | 16 | 0 | 4 | 0 | 20 | ✅ | | Data Stores | 4 | 0 | 0 | 0 | 4 | ✅ | | Plan | 8 | 0 | 0 | 0 | 8 | ✅ | | Govern | 6 | 0 | 0 | 0 | 6 | ✅ | | Package | 0 | 0 | 2 | 0 | 2 | ➖ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 58 | 0 | 14 | 0 | 72 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+
mentioned in issue #435259 (closed)
- Resolved by Max Woolf
- A deleted user
added documentation label
- A deleted user
added feature flag label
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@e47da6f8
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@16fb8b6f
- Resolved by Max Woolf
So I had a bit more of a play with the prompt in fb8aaa6b. Here's some example inputs and outputs.
Example 1
Input
How many people used the application in the last 7 days split up by operating system?
Output
{ "measures": [ "TrackedEvents.uniqueUsersCount" ], "dimensions": [ "TrackedEvents.osFamily" ], "timeDimensions": [ { "dimension": "TrackedEvents.derivedTstamp", "dateRange": "last 7 days", "granularity": "day" } ] }
Example 2
Input
How many total hits did the application every minute split up by browser family?
Ouptut
{ "measures": [ "TrackedEvents.pageViewsCount" ], "dimensions": [ "TrackedEvents.agentFamily", "TrackedEvents.derivedTstamp" ], "timeDimensions": [ { "dimension": "TrackedEvents.derivedTstamp", "dateRange": "last week", "granularity": "minute" } ], "order": { "TrackedEvents.derivedTstamp": "asc" } }
I need to test them out, but they look fairly good. I think. But it does have the issue, as @elwyn-gitlab quite rightly pointed out, that the prompt is essentially another SSoT that needs maintaining, which is a pain but I wonder for an initial iteration whether this is acceptable.
I'd like your thoughts though @elwyn-gitlab because although we have short toes; I'm wary of stepping on them
Edited by Max Woolf
- Resolved by Max Woolf
Hey @mwoolf, I'm hooking this MR up to the frontend here, but not getting any response back from the
aiCompletionResponse
subscription. I can see thegenerateCubeQuery
mutation go through just fine, and tailing thellm.log
I can see it correctly generates a cube query, but the subscription never gets data back. I can't spot anything different in my component compared to other components using theaiCompletionResponse
subscription. I tried adding aclientSubscriptionId
which I see some things using, but no joy - wondering if you have any ideas?Edited by Elwyn Benson
changed milestone to %16.8
- Resolved by Nicolas Dular
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@46f1d88c
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@e7cef782
- Resolved by Nicolas Dular
@elwyn-gitlab This MR is basically ready for review, I think. Were you able to resolve the issues getting it working? Let me know if you need some more assistance!
- Resolved by Lorena Ciutacu
@lciutacu Could you review the documentation changes please?
requested review from @lciutacu
@halilcoban @mksionek Could you both perform backend review please? Gosia, if you could focus on making sure I've implemented the AI framework correctly (I'm sure I haven't!) and Halil on a general backend review, I'd really appreciate it. Thanks
requested review from @halilcoban and @mksionek
8 Warnings de201ca8: The commit subject may not be longer than 72 characters. For more information, take a look at our Commit message guidelines. de201ca8: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. b7c6fdc0: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 2783a89f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 05613fa3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 55049c3d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 55049c3d: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 7ad5e875: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 2 Messages This MR contains docs in the /development directory. Any Maintainer, other than the author, can merge. You do not need tech writer review. This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/api/graphql/reference/index.md
(Link to current live version) -
doc/user/ai_features.md
(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend @rkumar555
(UTC+1, 1 hour ahead of author)
@dstull
(UTC-5, 5 hours behind author)
~"Authorization" Reviewer review is optional for ~"Authorization" @alexbuijs
(UTC+1, 1 hour ahead of author)
Please check reviewer's status!
Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger-
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@f48aeaf4
- Resolved by Elwyn Benson
mentioned in merge request !140277 (merged)
- Resolved by Elwyn Benson
- Resolved by Elwyn Benson
- Resolved by Lorena Ciutacu
- Resolved by Lorena Ciutacu
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@e3c021cf
Generated bygitlab_quality-test_tooling
.
Slow tests detected in this merge request. These slow tests might be related to this merge request's changes.Click to expand
Job File Name Duration Expected duration #5881638622 spec/features/groups/labels/subscription_spec.rb#L17
Labels subscription when signed in users can subscribe/unsubscribe to group labels 99.85 s < 50.13 s #5881641359 spec/features/groups/labels/subscription_spec.rb#L17
Labels subscription when signed in users can subscribe/unsubscribe to group labels 65.19 s < 50.13 s #5882442468 spec/features/groups/labels/subscription_spec.rb#L17
Labels subscription when signed in users can subscribe/unsubscribe to group labels 65.58 s < 50.13 s - A deleted user
added rspec:slow test detected label
added Technical Writing label
removed review request for @lciutacu
- Resolved by Nicolas Dular
@lciutacu
, thanks for approving this merge request.This is the first time the merge request has been approved. To ensure we don't only run predictive pipelines, and we don't break
master
, a new pipeline will be started shortly.Please wait for the pipeline to start before resolving this discussion and set auto-merge for the new pipeline. See merging a merge request for more details.
added pipeline:mr-approved label
Hi
@lciutacu
,GitLab Bot has added the Technical Writing label because a Technical Writer has approved or merged this MR.
This message was generated automatically. You're welcome to improve it.
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@3f578c7e
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@c7592cff
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@aa65b1e2
- Resolved by Nicolas Dular
- Resolved by Nicolas Dular
@mwoolf I will reassign to @nicolasdular as he has less load and the same amount of experience with AI
requested review from @nicolasdular and removed review request for @mksionek
- Resolved by Nicolas Dular
- Resolved by Nicolas Dular
- Resolved by Nicolas Dular
- Resolved by Nicolas Dular
Thanks @nicolasdular and @halilcoban! I've responded to your comments. Back to you both
- A deleted user
added development guidelines docsimprovement maintenancerefactor typemaintenance labels and removed typefeature label
removed featureaddition label
removed review request for @halilcoban
added 853 commits
-
b7c6fdc0...6e85d4e4 - 852 commits from branch
master
- de201ca8 - Merge remote-tracking branch 'origin/master' into mw/llm-product-analytics-poc
-
b7c6fdc0...6e85d4e4 - 852 commits from branch
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@de912a11
added 1 commit
- 71077d92 - Refactor project policy to use licenced flag
mentioned in merge request !141389 (merged)
mentioned in issue #435262
8 Warnings de201ca8: The commit subject may not be longer than 72 characters. For more information, take a look at our Commit message guidelines. de201ca8: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. b7c6fdc0: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 2783a89f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 05613fa3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 55049c3d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 55049c3d: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 7ad5e875: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 2 Messages This MR contains docs in the /development directory. Any Maintainer, other than the author, can merge. You do not need tech writer review. This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/api/graphql/reference/index.md
(Link to current live version) -
doc/user/ai_features.md
(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend @alexbuijs
(UTC+1, 1 hour ahead of author)
@alejandro
(UTC-3, 3 hours behind author)
~"Authorization" Reviewer review is optional for ~"Authorization" @jarka
(UTC+1, 1 hour ahead of author)
Please check reviewer's status!
Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger-
added 774 commits
-
b521912b...da8fe85d - 773 commits from branch
master
- c3de9717 - Merge branch 'master' into 'mw/llm-product-analytics-poc'
-
b521912b...da8fe85d - 773 commits from branch
- Resolved by Max Woolf
Hi @nicolasdular, This MR should be ready for your final review now please. Apologies for the delay
changed milestone to %16.9
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@872260b8
requested review from @nicolasdular
enabled an automatic merge when all merge checks for 872260b8 pass
added 147 commits
-
34c1e2f3...104a66a2 - 146 commits from branch
master
- 5a1c8c84 - Merge branch 'master' into 'mw/llm-product-analytics-poc'
-
34c1e2f3...104a66a2 - 146 commits from branch
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@947741ca
- Resolved by Nicolas Dular
Hey @alexbuijs
Seems like we need apolicies/*
approval as well - would you mind to have a look at this small change please so we can merge the MR? :)
requested review from @alexbuijs
removed review request for @alexbuijs
mentioned in commit 55b51664
mentioned in incident gitlab-org/quality/engineering-productivity/master-broken-incidents#4944 (closed)
added workflowstaging-canary label
@mwoolf This merge request was deployed to the workflowstaging-canary environment. You may want to enable the associated feature flag on this environment with/chatops run feature set generate_cube_query true --staging
.This message was generated automatically. You're welcome to improve it.
@mwoolf This merge request was deployed to the workflowcanary environment. You may want to enable the associated feature flag on this environment with/chatops run feature set generate_cube_query true --production
.This message was generated automatically. You're welcome to improve it.
@mwoolf This merge request was deployed to the workflowstaging environment. You may want to enable the associated feature flag on this environment with/chatops run feature set generate_cube_query true --staging
.This message was generated automatically. You're welcome to improve it.
@mwoolf This merge request was deployed to the workflowproduction environment. You may want to enable the associated feature flag on this environment with/chatops run feature set generate_cube_query true --production
.This message was generated automatically. You're welcome to improve it.
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
added releasedcandidate label
mentioned in issue gitlab-org/quality/triage-reports#17481 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17940 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18480 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18960 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19410 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20638 (closed)
mentioned in issue gitlab-org/quality/triage-reports#21001 (closed)
mentioned in issue gitlab-org/quality/triage-reports#21567 (closed)
mentioned in issue gitlab-org/quality/triage-reports#22060