Skip to content
Snippets Groups Projects

Generate a cube query using Vertex LLM

Merged Max Woolf requested to merge mw/llm-product-analytics-poc into master
1 unresolved thread

What does this MR do and why?

  • Adds a new AI Action :generate_cube_query that takes a question and returns a query that is a valid Cube query using Cube query format.

Things this MR is missing

  • Specs
  • Battle-testing the LLM prompt
  • Authorization, currently anybody can use it.
  • Prompt injection protection against access to other projects' data
  • ...probably other things

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  • Check out branch
  • Enable AI features locally: https://docs.gitlab.com/ee/development/ai_features/#test-ai-features-locally (You might be able to skip the embedding database, I'm not sure)
  • Make sure you've enabled Vertex access and configured it with your GDK. This action does not use Anthropic's API, only Vertex.
  • Enable the :expanded_ai_logging feature flag locally.
  • Enable the :generate_cube_query flag locally. NEW
  • Update the code here from perform_async to perform_inline. (This just makes testing easier, don't commit it anywhere.)
  • In another terminal, tail the LLM logs in your GDK: tail -f log/llm.log
  • Make a GraphQL call like this - feel free to update the question parameter:
mutation {
  aiAction(input: {generateCubeQuery: {question: "How many unique users used the application this week?", resourceId: "gid://gitlab/Project/19"}, clientSubscriptionId: "ARandomID"}) {
    clientMutationId
    errors 
  }
}
  • Assuming everything was set up correctly, you should (after a second or two), see a response from Vertex with a cube query :point_down: This will also be send to any open GraphQL subscriptions, but looking at the logs is a quicker way to see that it's worked.

Screenshot_2023-12-18_at_16.30.21

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Woolf

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Max Woolf changed the description

    changed the description

  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

  • A deleted user added documentation label

    added documentation label

  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

  • A deleted user added feature flag label

    added feature flag label

  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

    • Author Maintainer
      Resolved by Max Woolf

      So I had a bit more of a play with the prompt in fb8aaa6b. Here's some example inputs and outputs.

      Example 1

      Input

      How many people used the application in the last 7 days split up by operating system?

      Output

      {
        "measures": [
          "TrackedEvents.uniqueUsersCount"
        ],
        "dimensions": [
          "TrackedEvents.osFamily"
        ],
        "timeDimensions": [
          {
            "dimension": "TrackedEvents.derivedTstamp",
            "dateRange": "last 7 days",
            "granularity": "day"
          }
        ]
      }

      Example 2

      Input

      How many total hits did the application every minute split up by browser family?

      Ouptut

      {
        "measures": [
          "TrackedEvents.pageViewsCount"
        ],
        "dimensions": [
          "TrackedEvents.agentFamily",
          "TrackedEvents.derivedTstamp"
        ],
        "timeDimensions": [
          {
            "dimension": "TrackedEvents.derivedTstamp",
            "dateRange": "last week",
            "granularity": "minute"
          }
        ],
        "order": {
          "TrackedEvents.derivedTstamp": "asc"
        }
      }

      I need to test them out, but they look fairly good. I think. But it does have the issue, as @elwyn-gitlab quite rightly pointed out, that the prompt is essentially another SSoT that needs maintaining, which is a pain but I wonder for an initial iteration whether this is acceptable.

      I'd like your thoughts though @elwyn-gitlab because although we have short toes; I'm wary of stepping on them :smile_cat:

      Edited by Max Woolf
    • Resolved by Max Woolf

      Hey @mwoolf, I'm hooking this MR up to the frontend here, but not getting any response back from the aiCompletionResponse subscription. I can see the generateCubeQuery mutation go through just fine, and tailing the llm.log I can see it correctly generates a cube query, but the subscription never gets data back. I can't spot anything different in my component compared to other components using the aiCompletionResponse subscription. I tried adding a clientSubscriptionId which I see some things using, but no joy - wondering if you have any ideas?

      Edited by Elwyn Benson
  • Max Woolf changed milestone to %16.8

    changed milestone to %16.8

  • Max Woolf marked the checklist item Specs as completed

    marked the checklist item Specs as completed

  • Max Woolf marked the checklist item Authorization, currently anybody can use it. as completed

    marked the checklist item Authorization, currently anybody can use it. as completed

  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

  • Max Woolf marked the checklist item Battle-testing the LLM prompt as completed

    marked the checklist item Battle-testing the LLM prompt as completed

  • Max Woolf marked the checklist item Prompt injection protection against access to other projects' data as completed

    marked the checklist item Prompt injection protection against access to other projects' data as completed

  • Max Woolf marked the checklist item ...probably other things as completed

    marked the checklist item ...probably other things as completed

  • Max Woolf marked this merge request as ready

    marked this merge request as ready

  • Max Woolf resolved all threads

    resolved all threads

  • Max Woolf changed title from Draft PoC for product analytics AI use-case to Generate a cube query using Vertex LLM

    changed title from Draft PoC for product analytics AI use-case to Generate a cube query using Vertex LLM

  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

  • Max Woolf requested review from @lciutacu

    requested review from @lciutacu

  • Author Maintainer

    @halilcoban @mksionek Could you both perform backend review please? Gosia, if you could focus on making sure I've implemented the AI framework correctly (I'm sure I haven't!) and Halil on a general backend review, I'd really appreciate it. Thanks :pray:

  • Max Woolf requested review from @halilcoban and @mksionek

    requested review from @halilcoban and @mksionek

  • Max Woolf added 1955 commits

    added 1955 commits

    Compare with previous version

  • Contributor
    8 Warnings
    :warning: de201ca8: The commit subject may not be longer than 72 characters. For more information, take a look at our Commit message guidelines.
    :warning: de201ca8: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: b7c6fdc0: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 2783a89f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 05613fa3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 55049c3d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 55049c3d: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 7ad5e875: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    2 Messages
    :book: This MR contains docs in the /development directory. Any Maintainer, other than the author, can merge. You do not need tech writer review.
    :book: This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

    Documentation review

    The following files require a review from a technical writer:

    The review does not need to block merging this merge request. See the:

    Reviewer roulette

    Changes that require review have been detected!

    Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

    Category Reviewer Maintainer
    backend @rkumar555 profile link current availability (UTC+1, 1 hour ahead of author) @dstull profile link current availability (UTC-5, 5 hours behind author)
    ~"Authorization" Reviewer review is optional for ~"Authorization" @alexbuijs profile link current availability (UTC+1, 1 hour ahead of author)

    Please check reviewer's status!

    • available Reviewer is available!
    • unavailable Reviewer is unavailable!

    Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

    To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

    Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

  • Elwyn Benson
  • Elwyn Benson mentioned in merge request !140277 (merged)

    mentioned in merge request !140277 (merged)

  • Elwyn Benson
  • Lorena Ciutacu
  • Max Woolf added 1 commit

    added 1 commit

    • c6b7d056 - Documentation review changes

    Compare with previous version

  • Contributor

    :tools: Generated by gitlab_quality-test_tooling.


    :snail: Slow tests detected in this merge request. These slow tests might be related to this merge request's changes.

    Click to expand
    Job File Name Duration Expected duration
    #5881638622 spec/features/groups/labels/subscription_spec.rb#L17 Labels subscription when signed in users can subscribe/unsubscribe to group labels 99.85 s < 50.13 s
    #5881641359 spec/features/groups/labels/subscription_spec.rb#L17 Labels subscription when signed in users can subscribe/unsubscribe to group labels 65.19 s < 50.13 s
    #5882442468 spec/features/groups/labels/subscription_spec.rb#L17 Labels subscription when signed in users can subscribe/unsubscribe to group labels 65.58 s < 50.13 s
  • A deleted user added rspec:slow test detected label
  • Lorena Ciutacu approved this merge request

    approved this merge request

  • Lorena Ciutacu removed review request for @lciutacu

    removed review request for @lciutacu

  • Hi @lciutacu :wave:,

    GitLab Bot has added the Technical Writing label because a Technical Writer has approved or merged this MR.

    This message was generated automatically. You're welcome to improve it.

  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

  • Max Woolf added 1 commit

    added 1 commit

    • 3486527a - Remove markdown tags from vertex output

    Compare with previous version

  • Halil Coban
  • Gosia Ksionek requested review from @nicolasdular and removed review request for @mksionek

    requested review from @nicolasdular and removed review request for @mksionek

  • Nicolas Dular
  • Nicolas Dular
  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

  • Halil Coban approved this merge request

    approved this merge request

  • Halil Coban removed review request for @halilcoban

    removed review request for @halilcoban

  • Max Woolf added 853 commits

    added 853 commits

    • b7c6fdc0...6e85d4e4 - 852 commits from branch master
    • de201ca8 - Merge remote-tracking branch 'origin/master' into mw/llm-product-analytics-poc

    Compare with previous version

  • Max Woolf added 1 commit

    added 1 commit

    • 71077d92 - Refactor project policy to use licenced flag

    Compare with previous version

  • Nicolas Dular mentioned in merge request !141389 (merged)

    mentioned in merge request !141389 (merged)

  • Elwyn Benson mentioned in issue #435262

    mentioned in issue #435262

  • Max Woolf added 1 commit

    added 1 commit

    Compare with previous version

  • Contributor
    8 Warnings
    :warning: de201ca8: The commit subject may not be longer than 72 characters. For more information, take a look at our Commit message guidelines.
    :warning: de201ca8: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: b7c6fdc0: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 2783a89f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 05613fa3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 55049c3d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 55049c3d: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 7ad5e875: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    2 Messages
    :book: This MR contains docs in the /development directory. Any Maintainer, other than the author, can merge. You do not need tech writer review.
    :book: This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

    Documentation review

    The following files require a review from a technical writer:

    The review does not need to block merging this merge request. See the:

    Reviewer roulette

    Changes that require review have been detected!

    Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

    Category Reviewer Maintainer
    backend @alexbuijs profile link current availability (UTC+1, 1 hour ahead of author) @alejandro profile link current availability (UTC-3, 3 hours behind author)
    ~"Authorization" Reviewer review is optional for ~"Authorization" @jarka profile link current availability (UTC+1, 1 hour ahead of author)

    Please check reviewer's status!

    • available Reviewer is available!
    • unavailable Reviewer is unavailable!

    Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

    To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

    Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

  • Max Woolf added 774 commits

    added 774 commits

    Compare with previous version

  • Max Woolf
  • Max Woolf added 1 commit

    added 1 commit

    • 34c1e2f3 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Author Maintainer

    Hi @nicolasdular, This MR should be ready for your final review now please. Apologies for the delay :smile_cat:

  • Max Woolf changed milestone to %16.9

    changed milestone to %16.9

  • Nicolas Dular requested review from @nicolasdular

    requested review from @nicolasdular

  • Nicolas Dular approved this merge request

    approved this merge request

  • Nicolas Dular enabled an automatic merge when all merge checks for 872260b8 pass

    enabled an automatic merge when all merge checks for 872260b8 pass

  • Nicolas Dular resolved all threads

    resolved all threads

  • Max Woolf aborted the automatic merge because source branch was updated

    aborted the automatic merge because source branch was updated

  • Max Woolf added 147 commits

    added 147 commits

    Compare with previous version

  • Nicolas Dular approved this merge request

    approved this merge request

  • Nicolas Dular requested review from @alexbuijs

    requested review from @alexbuijs

  • Alex Buijs approved this merge request

    approved this merge request

  • Alex Buijs removed review request for @alexbuijs

    removed review request for @alexbuijs

  • Nicolas Dular resolved all threads

    resolved all threads

  • merged

  • Nicolas Dular mentioned in commit 55b51664

    mentioned in commit 55b51664

  • added workflowstaging label and removed workflowcanary label

  • Please register or sign in to reply
    Loading