Skip to content
Snippets Groups Projects

Add Get Web hook events API

2 unresolved threads

What does this MR do and why?

Add get web hook events on group and project by REST API

Connected to #437188 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Create web hook event
  2. Request to API
curl --request GET \
  --url http://gdk.test:3000/api/v4/<groups|projects>/<PARENT_ID>/hooks/<HOOK_ID>/events \
  --header 'PRIVATE-TOKEN: <TOKEN>' \

Example query plan from WebHooks::WebHookLogsFinder

Query plans

Example of use:

hook = WebHook.first
user = User.first

WebHooks::WebHookLogsFinder.new(hook, user, hook.parent, { status: ['client_failure', 200, 500] }).execute

Example of raw SQL (8075115 is a webhook ID owned by gitlab-org/gitlab):

SELECT "web_hook_logs".*
FROM   "web_hook_logs"
WHERE  "web_hook_logs"."web_hook_id" = 8075115
       AND ( "web_hook_logs"."response_status" BETWEEN '400' AND '499'
              OR "web_hook_logs"."response_status" = '200'
              OR "web_hook_logs"."response_status" = '500' ) 

Example of query plan:

# In postgres.ai, first create the index being added in this MR:
exec CREATE INDEX web_hook_logs_on_web_hook_id_and_response_status ON ONLY web_hook_logs USING btree (web_hook_id, response_status);

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/28697/commands/89415

 Append  (cost=0.57..46247.79 rows=12050 width=1382) (actual time=11.559..111781.689 rows=77803 loops=1)
   Buffers: shared hit=27 read=78197 dirtied=1642
   I/O Timings: read=110568.641 write=0.000
   ->  Index Scan using web_hook_logs_202402_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202402 web_hook_logs_1  (cost=0.57..37136.24 rows=9470 width=1389) (actual time=11.557..30403.693 rows=22217 loops=1)
         Index Cond: (web_hook_logs_1.web_hook_id = 8075115)
         Filter: ((((web_hook_logs_1.response_status)::text >= '400'::text) AND ((web_hook_logs_1.response_status)::text <= '499'::text)) OR ((web_hook_logs_1.response_status)::text = '200'::text) OR ((web_hook_logs_1.response_status)::text = '500'::text))
         Rows Removed by Filter: 4
         Buffers: shared hit=5 read=22318 dirtied=6
         I/O Timings: read=30217.506 write=0.000
   ->  Index Scan using web_hook_logs_202403_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202403 web_hook_logs_2  (cost=0.57..2882.54 rows=741 width=1369) (actual time=8.838..21558.973 rows=15551 loops=1)
         Index Cond: (web_hook_logs_2.web_hook_id = 8075115)
         Filter: ((((web_hook_logs_2.response_status)::text >= '400'::text) AND ((web_hook_logs_2.response_status)::text <= '499'::text)) OR ((web_hook_logs_2.response_status)::text = '200'::text) OR ((web_hook_logs_2.response_status)::text = '500'::text))
         Rows Removed by Filter: 1
         Buffers: shared hit=10 read=15622 dirtied=15
         I/O Timings: read=21425.277 write=0.000
   ->  Index Scan using web_hook_logs_202404_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202404 web_hook_logs_3  (cost=0.57..3141.61 rows=859 width=1368) (actual time=11.327..29921.386 rows=19370 loops=1)
         Index Cond: (web_hook_logs_3.web_hook_id = 8075115)
         Filter: ((((web_hook_logs_3.response_status)::text >= '400'::text) AND ((web_hook_logs_3.response_status)::text <= '499'::text)) OR ((web_hook_logs_3.response_status)::text = '200'::text) OR ((web_hook_logs_3.response_status)::text = '500'::text))
         Rows Removed by Filter: 6
         Buffers: shared hit=3 read=19486 dirtied=20
         I/O Timings: read=29730.146 write=0.000
   ->  Index Scan using web_hook_logs_202405_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202405 web_hook_logs_4  (cost=0.57..3027.14 rows=974 width=1347) (actual time=9.138..29854.263 rows=20665 loops=1)
         Index Cond: (web_hook_logs_4.web_hook_id = 8075115)
         Filter: ((((web_hook_logs_4.response_status)::text >= '400'::text) AND ((web_hook_logs_4.response_status)::text <= '499'::text)) OR ((web_hook_logs_4.response_status)::text = '200'::text) OR ((web_hook_logs_4.response_status)::text = '500'::text))
         Rows Removed by Filter: 17
         Buffers: shared hit=9 read=20771 dirtied=1601
         I/O Timings: read=29195.713 write=0.000
   ->  Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202406 web_hook_logs_5  (cost=0.00..0.00 rows=1 width=324) (actual time=0.015..0.015 rows=0 loops=1)
         Filter: ((web_hook_logs_5.web_hook_id = 8075115) AND ((((web_hook_logs_5.response_status)::text >= '400'::text) AND ((web_hook_logs_5.response_status)::text <= '499'::text)) OR ((web_hook_logs_5.response_status)::text = '200'::text) OR ((web_hook_logs_5.response_status)::text = '500'::text)))
         Rows Removed by Filter: 0
         I/O Timings: read=0.000 write=0.000
   ->  Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202407 web_hook_logs_6  (cost=0.00..0.00 rows=1 width=324) (actual time=0.008..0.008 rows=0 loops=1)
         Filter: ((web_hook_logs_6.web_hook_id = 8075115) AND ((((web_hook_logs_6.response_status)::text >= '400'::text) AND ((web_hook_logs_6.response_status)::text <= '499'::text)) OR ((web_hook_logs_6.response_status)::text = '200'::text) OR ((web_hook_logs_6.response_status)::text = '500'::text)))
         Rows Removed by Filter: 0
         I/O Timings: read=0.000 write=0.000
   ->  Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202408 web_hook_logs_7  (cost=0.00..0.00 rows=1 width=324) (actual time=0.007..0.007 rows=0 loops=1)
         Filter: ((web_hook_logs_7.web_hook_id = 8075115) AND ((((web_hook_logs_7.response_status)::text >= '400'::text) AND ((web_hook_logs_7.response_status)::text <= '499'::text)) OR ((web_hook_logs_7.response_status)::text = '200'::text) OR ((web_hook_logs_7.response_status)::text = '500'::text)))
         Rows Removed by Filter: 0
         I/O Timings: read=0.000 write=0.000
   ->  Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202409 web_hook_logs_8  (cost=0.00..0.00 rows=1 width=324) (actual time=0.006..0.006 rows=0 loops=1)
         Filter: ((web_hook_logs_8.web_hook_id = 8075115) AND ((((web_hook_logs_8.response_status)::text >= '400'::text) AND ((web_hook_logs_8.response_status)::text <= '499'::text)) OR ((web_hook_logs_8.response_status)::text = '200'::text) OR ((web_hook_logs_8.response_status)::text = '500'::text)))
         Rows Removed by Filter: 0
         I/O Timings: read=0.000 write=0.000
   ->  Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202410 web_hook_logs_9  (cost=0.00..0.00 rows=1 width=324) (actual time=0.007..0.007 rows=0 loops=1)
         Filter: ((web_hook_logs_9.web_hook_id = 8075115) AND ((((web_hook_logs_9.response_status)::text >= '400'::text) AND ((web_hook_logs_9.response_status)::text <= '499'::text)) OR ((web_hook_logs_9.response_status)::text = '200'::text) OR ((web_hook_logs_9.response_status)::text = '500'::text)))
         Rows Removed by Filter: 0
         I/O Timings: read=0.000 write=0.000
   ->  Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202411 web_hook_logs_10  (cost=0.00..0.00 rows=1 width=324) (actual time=0.012..0.012 rows=0 loops=1)
         Filter: ((web_hook_logs_10.web_hook_id = 8075115) AND ((((web_hook_logs_10.response_status)::text >= '400'::text) AND ((web_hook_logs_10.response_status)::text <= '499'::text)) OR ((web_hook_logs_10.response_status)::text = '200'::text) OR ((web_hook_logs_10.response_status)::text = '500'::text)))
         Rows Removed by Filter: 0
         I/O Timings: read=0.000 write=0.000
Edited by Jarka Košanová

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • mentioned in merge request !150486 (merged)

  • requested review from @.luke

  • Luke Duncalfe changed milestone to %17.0

    changed milestone to %17.0

  • 9 Warnings
    :warning: This MR changes code in ee/, but its Changelog commit is missing the EE: true trailer. Consider adding it to your Changelog commits.
    :warning: This merge request is quite big (1099 lines changed), please consider splitting it into multiple merge requests.
    :warning: 49e56a95: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 49e56a95: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 45282c1f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 45282c1f: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 3774af1d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: afcf29f3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: afcf29f3: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    1 Message
    :book: This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

    Documentation review

    The following files require a review from a technical writer:

    The review does not need to block merging this merge request. See the:

    Reviewer roulette

    Category Reviewer Maintainer
    backend @ddieulivol profile link current availability (UTC+2) @tianwenchen profile link current availability (UTC+10)
    database @carlad-gl profile link current availability (UTC+2) @jon_jenkins profile link current availability (UTC-5)
    groupimport and integrate (backend) @SamWord profile link current availability (UTC-4) Maintainer review is optional for groupimport and integrate (backend)

    Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

  • Luke Duncalfe
  • Luke Duncalfe requested review from @bhrai

    requested review from @bhrai

  • @lifez Thank you, this direction looks good to me :bow:. I've asked a database review for some early feedback. Besides whatever feedback comes from that early database review, we'd also want (you probably intend to do these, but I wanted to mention):

    Thank you :heart:. Please @gitlab-bot ready @.luke when you're ready :bow:

  • Luke Duncalfe removed review request for @.luke

    removed review request for @.luke

  • Bishwa Hang Rai removed review request for @bhrai

    removed review request for @bhrai

  • added 5997 commits

    Compare with previous version

  • 9 Warnings
    :warning: This MR changes code in ee/, but its Changelog commit is missing the EE: true trailer. Consider adding it to your Changelog commits.
    :warning: This merge request is quite big (1099 lines changed), please consider splitting it into multiple merge requests.
    :warning: 49e56a95: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 49e56a95: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 45282c1f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: 45282c1f: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    :warning: 3774af1d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: afcf29f3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
    :warning: afcf29f3: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
    1 Message
    :book: This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

    Documentation review

    The following files require a review from a technical writer:

    The review does not need to block merging this merge request. See the:

    Reviewer roulette

    Category Reviewer Maintainer
    backend @ddieulivol profile link current availability (UTC+2) @mc_rocha profile link current availability (UTC-4)
    database @jarka profile link current availability (UTC+2) @jon_jenkins profile link current availability (UTC-5)
    groupimport and integrate (backend) @SamWord profile link current availability (UTC-4) Maintainer review is optional for groupimport and integrate (backend)

    Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

    Edited by Danger Bot
  • added 527 commits

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • added 1 commit

    • ffb50dfa - Add web_hook_logs_finder_spec

    Compare with previous version

  • added 1 commit

    • 10a12fd3 - Add web_hook_logs_finder_spec

    Compare with previous version

  • added 1 commit

    • a7cfdb82 - Add web_hook_logs_finder_spec

    Compare with previous version

  • added 1 commit

    • c06e52d5 - Add web_hook_logs_finder_spec

    Compare with previous version

  • Phawin Khongkhasawan marked this merge request as ready

    marked this merge request as ready

  • Phawin Khongkhasawan changed the description

    changed the description

  • requested review from @.luke

  • Hi @lciutacu! Please review this documentation merge request. This message was generated automatically. You're welcome to improve it.

  • requested review from @lciutacu

  • @.luke @lciutacu, this Community contribution is ready for review.

    • Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
    • If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.

    This message was generated automatically. You're welcome to improve it.

  • Lorena Ciutacu removed review request for @lciutacu

    removed review request for @lciutacu

  • added 1 commit

    • 7150e5c3 - Apply 3 suggestion(s) to 2 file(s)

    Compare with previous version

  • Lorena Ciutacu approved this merge request

    approved this merge request

  • added pipelinetier-2 label and removed pipelinetier-1 label

  • Luke Duncalfe changed milestone to %17.1

    changed milestone to %17.1

  • Luke Duncalfe
  • Luke Duncalfe
  • Ghost User
  • Luke Duncalfe changed the description

    changed the description

  • Luke Duncalfe changed the description

    changed the description

  • E2E Test Result Summary

    allure-report-publisher generated test report!

    e2e-test-on-gdk: :white_check_mark: test report for 5c0d5a8d

    expand test summary
    +------------------------------------------------------------------+
    |                          suites summary                          |
    +-------------+--------+--------+---------+-------+-------+--------+
    |             | passed | failed | skipped | flaky | total | result |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Verify      | 44     | 0      | 3       | 0     | 47    | ✅     |
    | Create      | 128    | 0      | 12      | 0     | 140   | ✅     |
    | Package     | 19     | 0      | 12      | 0     | 31    | ✅     |
    | Data Stores | 31     | 0      | 0       | 0     | 31    | ✅     |
    | Govern      | 69     | 0      | 0       | 0     | 69    | ✅     |
    | Plan        | 67     | 0      | 2       | 0     | 69    | ✅     |
    | Monitor     | 8      | 0      | 0       | 0     | 8     | ✅     |
    | Analytics   | 2      | 0      | 0       | 0     | 2     | ✅     |
    | Manage      | 0      | 0      | 1       | 0     | 1     | ➖     |
    | Release     | 5      | 0      | 0       | 0     | 5     | ✅     |
    | Fulfillment | 1      | 0      | 0       | 0     | 1     | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Total       | 374    | 0      | 30      | 0     | 404   | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+

    e2e-package-and-test: :white_check_mark: test report for 5c0d5a8d

    expand test summary
    +-------------------------------------------------------------+
    |                       suites summary                        |
    +--------+--------+--------+---------+-------+-------+--------+
    |        | passed | failed | skipped | flaky | total | result |
    +--------+--------+--------+---------+-------+-------+--------+
    | Manage | 30     | 0      | 15      | 2     | 45    | ✅     |
    | Create | 274    | 0      | 34      | 40    | 308   | ✅     |
    | Plan   | 4      | 0      | 0       | 0     | 4     | ✅     |
    | Govern | 2      | 0      | 0       | 0     | 2     | ✅     |
    +--------+--------+--------+---------+-------+-------+--------+
    | Total  | 310    | 0      | 49      | 42    | 359   | ✅     |
    +--------+--------+--------+---------+-------+-------+--------+

    e2e-test-on-cng: :x: test report for e69d8234

    expand test summary
    +------------------------------------------------------------------+
    |                          suites summary                          |
    +-------------+--------+--------+---------+-------+-------+--------+
    |             | passed | failed | skipped | flaky | total | result |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Data Stores | 30     | 0      | 13      | 0     | 43    | ✅     |
    | Create      | 138    | 0      | 16      | 0     | 154   | ✅     |
    | Package     | 22     | 1      | 15      | 0     | 38    | ❌     |
    | Secure      | 3      | 0      | 2       | 0     | 5     | ✅     |
    | Plan        | 83     | 0      | 11      | 0     | 94    | ✅     |
    | Govern      | 78     | 1      | 9       | 1     | 88    | ❌     |
    | Verify      | 52     | 0      | 14      | 0     | 66    | ✅     |
    | Ai-powered  | 0      | 0      | 1       | 0     | 1     | ➖     |
    | Monitor     | 8      | 0      | 12      | 0     | 20    | ✅     |
    | Manage      | 2      | 0      | 8       | 0     | 10    | ✅     |
    | Analytics   | 2      | 0      | 0       | 0     | 2     | ✅     |
    | Configure   | 0      | 0      | 3       | 0     | 3     | ➖     |
    | Fulfillment | 2      | 0      | 8       | 0     | 10    | ✅     |
    | ModelOps    | 0      | 0      | 1       | 0     | 1     | ➖     |
    | Growth      | 0      | 0      | 2       | 0     | 2     | ➖     |
    | Release     | 5      | 0      | 1       | 0     | 6     | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Total       | 425    | 2      | 116     | 1     | 543   | ❌     |
    +-------------+--------+--------+---------+-------+-------+--------+
  • Luke Duncalfe
  • Database migrations (on the main database)

    6 Warnings
    :warning: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
    timing guidelines
    . Run time should not exceed 300000ms, but it was 2835566.06ms. Please consider
    possible options to improve the query performance.
    CREATE INDEX CONCURRENTLY
    "index_8c2627406d" ON "gitlab_partitions_dynamic"."web_hook_logs_202404" ("web_hook_id",
    "response_status")
    :warning: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
    timing guidelines
    . Run time should not exceed 300000ms, but it was 2626449.66ms. Please consider
    possible options to improve the query performance.
    CREATE INDEX CONCURRENTLY
    "index_8782333851" ON "gitlab_partitions_dynamic"."web_hook_logs_202405" ("web_hook_id",
    "response_status")
    :warning: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
    timing guidelines
    . Run time should not exceed 300000ms, but it was 2405871.16ms. Please consider
    possible options to improve the query performance.
    CREATE INDEX CONCURRENTLY
    "index_85b2fa997a" ON "gitlab_partitions_dynamic"."web_hook_logs_202403" ("web_hook_id",
    "response_status")
    :warning: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
    timing guidelines
    . Run time should not exceed 300000ms, but it was 2264890.57ms. Please consider
    possible options to improve the query performance.
    CREATE INDEX CONCURRENTLY
    "index_c2290bb536" ON "gitlab_partitions_dynamic"."web_hook_logs_202402" ("web_hook_id",
    "response_status")
    :warning: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId took 169.13min. Please add a
    comment that mentions Release Managers (@gitlab-org/release/managers) so they are informed.
    :warning: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId may need a batched
    background migration
    to comply with timing guidelines. It took 169.13min, but should not exceed 3.0min

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId Regular 10148.1 s :warning: +6.91 GiB
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 12
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 4

    :warning: Migration: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId

    • Type: Regular
    • Duration: 10148.1 s
    • Database size change: +6.91 GiB
    Calls Total Time Max Time Mean Time Rows Query
    1 2835566.1 ms 2835566.1 ms 2835566.1 ms 0
    CREATE INDEX CONCURRENTLY "index_8c2627406d" ON "gitlab_partitions_dynamic"."web_hook_logs_202404" ("web_hook_id", "response_status")
    1 2626449.7 ms 2626449.7 ms 2626449.7 ms 0
    CREATE INDEX CONCURRENTLY "index_8782333851" ON "gitlab_partitions_dynamic"."web_hook_logs_202405" ("web_hook_id", "response_status")
    1 2405871.2 ms 2405871.2 ms 2405871.2 ms 0
    CREATE INDEX CONCURRENTLY "index_85b2fa997a" ON "gitlab_partitions_dynamic"."web_hook_logs_202403" ("web_hook_id", "response_status")
    1 2264890.6 ms 2264890.6 ms 2264890.6 ms 0
    CREATE INDEX CONCURRENTLY "index_c2290bb536" ON "gitlab_partitions_dynamic"."web_hook_logs_202402" ("web_hook_id", "response_status")
    1 53.4 ms 53.4 ms 53.4 ms 0
    CREATE INDEX "web_hook_logs_on_web_hook_id_and_response_status" ON "web_hook_logs" ("web_hook_id", "response_status")
    1 39.2 ms 39.2 ms 39.2 ms 0
    CREATE INDEX CONCURRENTLY "index_b2c3e9b27b" ON "gitlab_partitions_dynamic"."web_hook_logs_202411" ("web_hook_id", "response_status")
    1 31.7 ms 31.7 ms 31.7 ms 0
    CREATE INDEX CONCURRENTLY "index_dd3617e333" ON "gitlab_partitions_dynamic"."web_hook_logs_202406" ("web_hook_id", "response_status")
    1 19.9 ms 19.9 ms 19.9 ms 0
    CREATE INDEX CONCURRENTLY "index_704bc80fa4" ON "gitlab_partitions_dynamic"."web_hook_logs_202408" ("web_hook_id", "response_status")
    1 11.9 ms 11.9 ms 11.9 ms 0
    CREATE INDEX CONCURRENTLY "index_9cfc33ea92" ON "gitlab_partitions_dynamic"."web_hook_logs_202407" ("web_hook_id", "response_status")
    1 11.2 ms 11.2 ms 11.2 ms 0
    CREATE INDEX CONCURRENTLY "index_e77f4400fb" ON "gitlab_partitions_dynamic"."web_hook_logs_202409" ("web_hook_id", "response_status")
    1 10.8 ms 10.8 ms 10.8 ms 0
    CREATE INDEX CONCURRENTLY "index_881be39fc6" ON "gitlab_partitions_dynamic"."web_hook_logs_202410" ("web_hook_id", "response_status")
    1 3.5 ms 3.5 ms 3.5 ms 10
    SELECT "postgres_partitions".*
    FROM "postgres_partitions" WHERE "postgres_partitions"."parent_identifier" = $1
    ORDER BY "postgres_partitions"."name" ASC
    1 0.9 ms 0.9 ms 0.9 ms 1
    SELECT "feature_gates"."key", "feature_gates"."value"  FROM "feature_gates"  WHERE "feature_gates"."feature_key" = $1
    1 0.3 ms 0.3 ms 0.3 ms 1
    SELECT "postgres_partitioned_tables".*
    FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
    LIMIT $3
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()

    :warning: Index creation timing exceeded

    These index creation queries might have caused your migration to exceed timing guidelines. Please examine the timing data on these queries and consider if the asynchronous index creation process is appropriate.

    Histogram for AddWebhookLogIndexOnResponseStatusAndWebHookId
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 12
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 4

    Other information

    No other migrations pending on GitLab.com

    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-3330508-14135626-main 2024-05-31T04:03:50Z 2024-05-31T03:09:58Z 2024-05-31 19:02:57 +0000
    database-testing-3330508-14135626-ci 2024-05-31T04:03:49Z 2024-05-31T03:45:22Z 2024-05-31 19:02:57 +0000

    Job artifacts

    Database migrations (on the ci database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId Regular 16.6 s :white_check_mark: +80.00 KiB
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 16
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId

    • Type: Regular
    • Duration: 16.6 s
    • Database size change: +80.00 KiB
    Calls Total Time Max Time Mean Time Rows Query
    1 15.0 ms 15.0 ms 15.0 ms 0
    CREATE INDEX "web_hook_logs_on_web_hook_id_and_response_status" ON "web_hook_logs" ("web_hook_id", "response_status")
    1 13.5 ms 13.5 ms 13.5 ms 0
    CREATE INDEX CONCURRENTLY "index_c2290bb536" ON "gitlab_partitions_dynamic"."web_hook_logs_202402" ("web_hook_id", "response_status")
    1 11.0 ms 11.0 ms 11.0 ms 0
    CREATE INDEX CONCURRENTLY "index_881be39fc6" ON "gitlab_partitions_dynamic"."web_hook_logs_202410" ("web_hook_id", "response_status")
    1 8.5 ms 8.5 ms 8.5 ms 0
    CREATE INDEX CONCURRENTLY "index_9cfc33ea92" ON "gitlab_partitions_dynamic"."web_hook_logs_202407" ("web_hook_id", "response_status")
    1 7.5 ms 7.5 ms 7.5 ms 0
    CREATE INDEX CONCURRENTLY "index_704bc80fa4" ON "gitlab_partitions_dynamic"."web_hook_logs_202408" ("web_hook_id", "response_status")
    1 7.4 ms 7.4 ms 7.4 ms 0
    CREATE INDEX CONCURRENTLY "index_dd3617e333" ON "gitlab_partitions_dynamic"."web_hook_logs_202406" ("web_hook_id", "response_status")
    1 7.1 ms 7.1 ms 7.1 ms 0
    CREATE INDEX CONCURRENTLY "index_8c2627406d" ON "gitlab_partitions_dynamic"."web_hook_logs_202404" ("web_hook_id", "response_status")
    1 6.6 ms 6.6 ms 6.6 ms 0
    CREATE INDEX CONCURRENTLY "index_8782333851" ON "gitlab_partitions_dynamic"."web_hook_logs_202405" ("web_hook_id", "response_status")
    1 6.1 ms 6.1 ms 6.1 ms 0
    CREATE INDEX CONCURRENTLY "index_e77f4400fb" ON "gitlab_partitions_dynamic"."web_hook_logs_202409" ("web_hook_id", "response_status")
    1 6.1 ms 6.1 ms 6.1 ms 0
    CREATE INDEX CONCURRENTLY "index_b2c3e9b27b" ON "gitlab_partitions_dynamic"."web_hook_logs_202411" ("web_hook_id", "response_status")
    1 6.0 ms 6.0 ms 6.0 ms 0
    CREATE INDEX CONCURRENTLY "index_85b2fa997a" ON "gitlab_partitions_dynamic"."web_hook_logs_202403" ("web_hook_id", "response_status")
    1 2.9 ms 2.9 ms 2.9 ms 10
    SELECT "postgres_partitions".*
    FROM "postgres_partitions" WHERE "postgres_partitions"."parent_identifier" = $1
    ORDER BY "postgres_partitions"."name" ASC
    1 0.2 ms 0.2 ms 0.2 ms 1
    SELECT "postgres_partitioned_tables".*
    FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
    LIMIT $3
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddWebhookLogIndexOnResponseStatusAndWebHookId
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 16
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    No other migrations pending on GitLab.com

    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-3330508-14135626-main 2024-05-31T04:03:50Z 2024-05-31T03:09:58Z 2024-05-31 19:02:57 +0000
    database-testing-3330508-14135626-ci 2024-05-31T04:03:49Z 2024-05-31T03:45:22Z 2024-05-31 19:02:57 +0000

    Job artifacts


    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

    Edited by ****
  • Luke Duncalfe requested review from @jarka

    requested review from @jarka

  • Luke Duncalfe removed review request for @.luke

    removed review request for @.luke

  • added 1 commit

    • 0d19ca34 - Apply 4 suggestion(s) to 2 file(s)

    Compare with previous version

  • Phawin Khongkhasawan reset approvals from @lciutacu by pushing to the branch

    reset approvals from @lciutacu by pushing to the branch

  • Jarka Košanová removed review request for @jarka

    removed review request for @jarka

  • added pipelinetier-1 label and removed pipelinetier-2 label

  • Hi @jarka :wave:

    We noticed this MR is marked as workflowready for review but no reviewer is assigned. workflowin dev has automatically been applied to this MR based on the likelihood the review is finished. If additional reviews are still required, please assign a reviewer and reapply workflowready for review.

    @lifez you may also request a review by commenting @gitlab-bot ready. You can also assign reviewers directly using @gitlab-bot ready @user1 @user2 if you know the relevant reviewer(s), such as those who were involved in a related issue.

    This message was generated automatically. You're welcome to improve it.

  • added 1 commit

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • requested review from @.luke

  • added 1 commit

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • added 1905 commits

    Compare with previous version

  • Luke Duncalfe
  • Luke Duncalfe
  • @lifez Thank you very much for all of your efforts here :heart: :muscle:! I appreciate all of your time here very much. I've left some new suggestions! Please @gitlab-bot ready @.luke when ready!

  • Luke Duncalfe removed review request for @.luke

    removed review request for @.luke

  • Hi @.luke :wave:

    We noticed this MR is marked as workflowready for review but no reviewer is assigned. workflowin dev has automatically been applied to this MR based on the likelihood the review is finished. If additional reviews are still required, please assign a reviewer and reapply workflowready for review.

    @lifez you may also request a review by commenting @gitlab-bot ready. You can also assign reviewers directly using @gitlab-bot ready @user1 @user2 if you know the relevant reviewer(s), such as those who were involved in a related issue.

    This message was generated automatically. You're welcome to improve it.

  • added 1 commit

    Compare with previous version

  • added 1741 commits

    Compare with previous version

  • requested review from @.luke

  • @.luke, this Community contribution is ready for review.

    • Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
    • If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.

    This message was generated automatically. You're welcome to improve it.

  • added 1 commit

    Compare with previous version

  • Luke Duncalfe
  • Luke Duncalfe
  • Luke Duncalfe
  • Luke Duncalfe removed review request for @.luke

    removed review request for @.luke

  • Luke Duncalfe changed milestone to %17.2

    changed milestone to %17.2

  • Luke Duncalfe requested review from @jarka

    requested review from @jarka

  • added 1 commit

    • 0dab3128 - Apply 3 suggestion(s) to 2 file(s)

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • requested review from @.luke

  • @jarka @.luke, this Community contribution is ready for review.

    • Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
    • If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.

    This message was generated automatically. You're welcome to improve it.

  • Evan Read requested changes

    requested changes

  • added 1 commit

    • 2150115f - Apply 2 suggestion(s) to 2 file(s)

    Compare with previous version

  • Evan Read requested review from @eread

    requested review from @eread

  • Ghost User
  • Evan Read approved this merge request

    approved this merge request

  • added pipelinetier-2 label and removed pipelinetier-1 label

    • Resolved by Luke Duncalfe

      Before you set this MR to auto-merge

      This merge request will progress on pipeline tiers until it reaches the last tier: pipelinetier-3.

      Before you resolve this discussion, please check the following:

      • You are the last maintainer of this merge request
      • The latest pipeline for this merge request is pipelinetier-3 (You can find which tier it is in the pipeline name)
      • This pipeline is recent enough (created in the last 8 hours)

      If all the criteria above apply, please resolve this discussion and the set auto-merge for this merge request.

      See pipeline tiers and merging a merge request for more details.

  • Evan Read removed review request for @eread

    removed review request for @eread

  • Luke Duncalfe approved this merge request

    approved this merge request

  • @lifez Thank you! I've left one suggestion, but this looks good to me so I'll approve now! There will be some changes to the finder, but I think the discussion is heading in the right direction, so I'll pass to a backend maintainer now.

    @jnutt Could you please give this Community contribution MR its backend maintainer review? :bow:

  • Luke Duncalfe requested review from @jnutt and removed review request for @.luke

    requested review from @jnutt and removed review request for @.luke

  • added 1 commit

    • a816cc08 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Phawin Khongkhasawan reset approvals from @.luke by pushing to the branch

    reset approvals from @.luke by pushing to the branch

  • added 1 commit

    • 62989894 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Phawin Khongkhasawan reset approvals from @eread by pushing to the branch

    reset approvals from @eread by pushing to the branch

  • added 1 commit

    • a3015f11 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • James Nutt
  • added 1 commit

    • f0a9a103 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • added 1 commit

    • 94ef12a3 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Evan Read approved this merge request

    approved this merge request

  • added 1 commit

    • 5a268bdd - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Jarka Košanová changed the description

    changed the description

  • Jarka Košanová requested review from @pshutsin and removed review request for @jarka

    requested review from @pshutsin and removed review request for @jarka

  • Pavel Shutsin
  • added 1 commit

    • aee13dbb - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • requested review from @pshutsin

  • added databaseapproved label and removed databasereviewed label

  • Pavel Shutsin approved this merge request

    approved this merge request

  • Pavel Shutsin removed review request for @pshutsin

    removed review request for @pshutsin

  • added pipelinetier-3 label and removed pipelinetier-2 label

  • added 2623 commits

    Compare with previous version

  • Phawin Khongkhasawan reset approvals from @eread and @pshutsin by pushing to the branch

    reset approvals from @eread and @pshutsin by pushing to the branch

  • added 1 commit

    • e45cb9d2 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • added 1 commit

    • 3c920b10 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • requested review from @.luke

  • James Nutt
  • added 1 commit

    • a91df484 - Apply 2 suggestion(s) to 2 file(s)

    Compare with previous version

  • added 1 commit

    • e69d8234 - Apply 2 suggestion(s) to 2 file(s)

    Compare with previous version

  • James Nutt approved this merge request

    approved this merge request

  • mentioned in merge request !151130 (merged)

  • Luke Duncalfe resolved all threads

    resolved all threads

  • Luke Duncalfe approved this merge request

    approved this merge request

    • Resolved by Luke Duncalfe

      @lifez Apologies for the delay :pray: thank you so much on all of your hard work here, I've really appreciated your collaborative attitude and all of the thoughtful conversations :heart:.

      We've lost the database approval that is required by our Codeowners rules which prevents me from merging yet. @pshutsin or @jarka could you please re-apply your approval :bow:

  • Luke Duncalfe requested review from @jarka and @pshutsin

    requested review from @jarka and @pshutsin

  • Evan Read approved this merge request

    approved this merge request

  • Jarka Košanová approved this merge request

    approved this merge request

  • Jarka Košanová removed review request for @jarka

    removed review request for @jarka

  • There's an undercoverage error https://gitlab.com/gitlab-org/gitlab/-/jobs/7302009284 since the pipelinetier-3 label was applied. I'm going to apply the pipeline:skip-undercoverage label because I worry we could get stuck in a cycle of waiting for approvals, pushing changes, and waiting for approvals again.

  • Pavel Shutsin resolved all threads

    resolved all threads

  • Pavel Shutsin approved this merge request

    approved this merge request

  • Pavel Shutsin removed review request for @pshutsin

    removed review request for @pshutsin

  • Luke Duncalfe started a merge train

    started a merge train

  • Luke Duncalfe removed this merge request from the merge train

    removed this merge request from the merge train

  • Luke Duncalfe enabled automatic add to merge train when checks pass

    enabled automatic add to merge train when checks pass

  • Luke Duncalfe resolved all threads

    resolved all threads

  • Luke Duncalfe cancelled automatic add to merge train

    cancelled automatic add to merge train

  • Luke Duncalfe
  • added 1 commit

    • c237c027 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Phawin Khongkhasawan reset approvals from @jarka, @.luke, @pshutsin, and @jnutt by pushing to the branch

    reset approvals from @jarka, @.luke, @pshutsin, and @jnutt by pushing to the branch

  • Luke Duncalfe requested review from @pshutsin

    requested review from @pshutsin

  • Luke Duncalfe approved this merge request

    approved this merge request

  • Pavel Shutsin approved this merge request

    approved this merge request

  • Pavel Shutsin removed review request for @pshutsin

    removed review request for @pshutsin

  • added 2586 commits

    Compare with previous version

  • Phawin Khongkhasawan reset approvals from @eread, @.luke, and @pshutsin by pushing to the branch

    reset approvals from @eread, @.luke, and @pshutsin by pushing to the branch

  • requested review from @pshutsin

  • Luke Duncalfe changed milestone to %17.3

    changed milestone to %17.3

  • Pavel Shutsin approved this merge request

    approved this merge request

  • Pavel Shutsin removed review request for @pshutsin

    removed review request for @pshutsin

  • Luke Duncalfe approved this merge request

    approved this merge request

  • Luke Duncalfe resolved all threads

    resolved all threads

  • requested review from @pshutsin

  • added 1 commit

    • 5c0d5a8d - Apply 2 suggestion(s) to 2 file(s)

    Compare with previous version

  • Phawin Khongkhasawan reset approvals from @.luke and @pshutsin by pushing to the branch

    reset approvals from @.luke and @pshutsin by pushing to the branch

  • Pavel Shutsin approved this merge request

    approved this merge request

  • Luke Duncalfe approved this merge request

    approved this merge request

  • Luke Duncalfe resolved all threads

    resolved all threads

  • :rocket: Let's merge! Thank you @lifez!

  • Luke Duncalfe enabled automatic add to merge train when checks pass

    enabled automatic add to merge train when checks pass

  • Luke Duncalfe started a merge train

    started a merge train

  • @lifez, how was your code review experience with this merge request? Please tell us how we can continue to iterate and improve:

    1. React with a :thumbsup: or a :thumbsdown: on this comment to describe your experience.
    2. Create a new comment starting with @gitlab-bot feedback below, and leave any additional feedback you have for us in the comment.

    Subscribe to the GitLab Community Newsletter for contributor-focused content and opportunities to level up.

    Thanks for your help! :heart:

    This message was generated automatically. You're welcome to improve it.

  • Hello @lifez :wave:

    The database team is looking for ways to improve the database review process and we would love your help!

    If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:

    @gitlab-org/database-team

    And someone will be by shortly!

    Thanks for your help! :heart:

    This message was generated automatically. You're welcome to improve it.

  • 30187 30187
    30188 30188 CREATE UNIQUE INDEX vulnerability_occurrence_pipelines_on_unique_keys ON vulnerability_occurrence_pipelines USING btree (occurrence_id, pipeline_id);
    30189 30189
    30190 CREATE INDEX web_hook_logs_on_web_hook_id_and_response_status ON ONLY web_hook_logs USING btree (web_hook_id, response_status);
    30191
    • @pshutsin This should not be included, as the index is not created yet. It should be added when the index is created synchronously.

    • @.luke Should we merge all this before the index is in place? :thinking:

    • @krasio Good catch, perhaps not. Do we need to remove this line?

    • Yes we need to remove it for now. I wonder should we have the rest without this index, I did not see a feature flag to guard it. How much of the feature is supported by the index?

    • @krasio @.luke I'm opening the MR to remove this index. Where do we put it back another MR?

      The index support the rest of these API. While we have recent guard. It shouldn't significantly impact performance.

    • Error: db/structure.sql was changed, and no migrations were added:
      diff --git a/db/structure.sql b/db/structure.sql
      index 403ff298643..8ac45e96e46 100644
      --- a/db/structure.sql
      +++ b/db/structure.sql
      @@ -30181,8 +30181,6 @@ CREATE UNIQUE INDEX virtual_registries_pkgs_maven_registries_on_unique_group_ids

      Got this error while commit. Should I skip the lefthook?

    • @lifez Yes I think so :thumbsup:

    • Edited by Phawin Khongkhasawan
    • @.luke @lifez Is this index actually helping? Do we have a plan proving this?

    • Here's a plan for the filter by status query, when this index is present. We can see it's not being used.

      gitlabhq_dblab=# explain (analyze , buffers ) SELECT
      gitlabhq_dblab-# "web_hook_logs".*
      gitlabhq_dblab-# FROM
      gitlabhq_dblab-# "web_hook_logs"
      gitlabhq_dblab-# WHERE
      gitlabhq_dblab-# "web_hook_logs"."web_hook_id" = 25401284
      gitlabhq_dblab-# AND "web_hook_logs"."response_status" = '400'
      gitlabhq_dblab-# AND "web_hook_logs"."created_at" BETWEEN '2024-07-12 00:00:00'
      gitlabhq_dblab-# AND '2024-07-19 00:42:17.342483'
      gitlabhq_dblab-# ORDER BY
      gitlabhq_dblab-# "web_hook_logs"."created_at" DESC
      gitlabhq_dblab-# LIMIT
      gitlabhq_dblab-# 20 OFFSET 0;
                                                                                                             QUERY PLAN                                                                                                        
      -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       Limit  (cost=0.57..15367.85 rows=20 width=1350) (actual time=131414.673..131414.686 rows=0 loops=1)
         Buffers: shared hit=38860 read=283066
         I/O Timings: read=128744.419
         ->  Index Scan Backward using web_hook_logs_202407_created_at_web_hook_id_idx on web_hook_logs_202407 web_hook_logs  (cost=0.57..3937865.87 rows=5125 width=1350) (actual time=131414.670..131414.671 rows=0 loops=1)
               Index Cond: ((created_at >= '2024-07-12 00:00:00'::timestamp without time zone) AND (created_at <= '2024-07-19 00:42:17.342483'::timestamp without time zone) AND (web_hook_id = 25401284))
               Filter: ((response_status)::text = '400'::text)
               Rows Removed by Filter: 45961
               Buffers: shared hit=38860 read=283066
               I/O Timings: read=128744.419
       Planning:
         Buffers: shared hit=14 read=5
         I/O Timings: read=8.785
       Planning Time: 10.862 ms
       Execution Time: 131414.766 ms
      (14 rows)
      
      Time: 131742.847 ms (02:11.743)
      gitlabhq_dblab=# \d web_hook_logs_on_web_hook_id_and_response_status
      Partitioned index "public.web_hook_logs_on_web_hook_id_and_response_status"
           Column      |       Type        | Key? |   Definition    
      -----------------+-------------------+------+-----------------
       web_hook_id     | integer           | yes  | web_hook_id
       response_status | character varying | yes  | response_status
      btree, for table "public.web_hook_logs"
      Number of partitions: 1 (Use \d+ to list them.)
    • @krasio Oh that's interesting. The query took 2 minutes to complete? So we're missing an index still, but we haven't managed to define one that PG will use, is that right? Would the correct index be on (web_hook_id, created_at, response_status) or perhaps (created_at, web_hook_id, response_status) I'm not sure which order is best.

      Edited by Luke Duncalfe
    • Index on (created_at, web_hook_id, response_status) helps, but it's still not ideal.

      Examples
      gitlabhq_dblab=# explain (analyze, buffers)
      SELECT
      	"web_hook_logs".*
      FROM
      	"web_hook_logs"
      WHERE
      	"web_hook_logs"."web_hook_id" = 25401284
      	AND "web_hook_logs"."response_status" = '400'
      	AND "web_hook_logs"."created_at" BETWEEN '2024-07-12 00:00:00'
      	AND '2024-07-19 00:42:17.342483'
      ORDER BY
      	"web_hook_logs"."created_at" DESC
      LIMIT
      	20 OFFSET 0;
                                                                                                                         QUERY PLAN                                                                                                                    
      -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       Limit  (cost=0.57..5005.63 rows=20 width=1350) (actual time=4973.807..4973.809 rows=0 loops=1)
         Buffers: shared hit=4 read=236947
         I/O Timings: read=3029.496
         ->  Index Scan Backward using index_1_202407 on web_hook_logs_202407 web_hook_logs  (cost=0.57..1282547.15 rows=5125 width=1350) (actual time=4973.803..4973.803 rows=0 loops=1)
               Index Cond: ((created_at >= '2024-07-12 00:00:00'::timestamp without time zone) AND (created_at <= '2024-07-19 00:42:17.342483'::timestamp without time zone) AND (web_hook_id = 25401284) AND ((response_status)::text = '400'::text))
               Buffers: shared hit=4 read=236947
               I/O Timings: read=3029.496
       Planning:
         Buffers: shared read=5
         I/O Timings: read=0.259
       Planning Time: 0.592 ms
       Execution Time: 4973.880 ms
      (12 rows)
      
      Time: 5282.178 ms (00:05.282)
      gitlabhq_dblab=# explain (analyze, buffers)
      SELECT
      	"web_hook_logs".*
      FROM
      	"web_hook_logs"
      WHERE
      	"web_hook_logs"."web_hook_id" = 25401284
      	AND "web_hook_logs"."response_status" BETWEEN '500'
      	AND '599'
      	AND "web_hook_logs"."created_at" BETWEEN '2024-07-12 00:00:00'
      	AND '2024-07-19 00:42:17.342483'
      ORDER BY
      	"web_hook_logs"."created_at" DESC
      LIMIT
      	20 OFFSET 0;
                                                                                                                                                QUERY PLAN                                                                                                                                           
      -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       Limit  (cost=0.57..6838.58 rows=20 width=1350) (actual time=4532.437..4532.439 rows=0 loops=1)
         Buffers: shared hit=4 read=236947
         I/O Timings: read=2581.960
         ->  Index Scan Backward using index_1_202407 on web_hook_logs_202407 web_hook_logs  (cost=0.57..1435297.89 rows=4198 width=1350) (actual time=4532.434..4532.435 rows=0 loops=1)
               Index Cond: ((created_at >= '2024-07-12 00:00:00'::timestamp without time zone) AND (created_at <= '2024-07-19 00:42:17.342483'::timestamp without time zone) AND (web_hook_id = 25401284) AND ((response_status)::text >= '500'::text) AND ((response_status)::text <= '599'::text))
               Buffers: shared hit=4 read=236947
               I/O Timings: read=2581.960
       Planning:
         Buffers: shared read=5
         I/O Timings: read=0.191
       Planning Time: 0.525 ms
       Execution Time: 4532.495 ms
      (12 rows)
      
      Time: 4872.137 ms (00:04.872)
      Edited by Krasimir Angelov
    • @.luke @lifez I think we should remove this new index, unless there is a proof it's helping.

    • @krasio Are these queries using the pagination limit of 20?

    • @.luke Yes. Sorry, the SQL was there but formatting was off so it was not visible. Fixed.

    • BTW web_hook_logs is one of the table we should be careful to add index to - https://gitlab.com/gitlab-org/gitlab/-/blob/84b753603028a2de854b50c3afbeaaf32e9c9b41/rubocop/cop/migration/prevent_index_creation.rb#L27. It was missed because the cop does not check for prepare_partitioned_async_index.

    • Please register or sign in to reply
  • mentioned in merge request !159862 (merged)

  • added workflowstaging label and removed workflowcanary label

  • Evan Read
  • Luke Duncalfe mentioned in merge request !160184 (merged)

    mentioned in merge request !160184 (merged)

  • Luke Duncalfe mentioned in commit b8130f74

    mentioned in commit b8130f74

  • Luke Duncalfe mentioned in merge request !162255 (merged)

    mentioned in merge request !162255 (merged)

  • Luke Duncalfe mentioned in commit dca0e967

    mentioned in commit dca0e967

    • Sorry to bother you guys. I saw this PR in the 17.3 release blog post and noticed that this community contribution is not available for 'free' users , neither SaaS or gitlab.com. That surprised me. Is this something new, is that an issue in the changelog or was it always possible to find community contributions behind the payed versions of gitlab?

    • @lexxxel It working fine on the project scope.

      curl --request GET \
        --url https://gitlab.com/api/v4/projects/60564468/hooks/43037580/events \
        --header 'PRIVATE-TOKEN: <PROJECT_ACCESS_TOKEN>' \

      On the other hand, group webhook are only available to Premium and Ultimate users.

      Btw, Should we consider further clarifying or separating the release post between Project and Group hooks? @m_frankiewicz

    • @lifez that makes sense. If the overlying feature is already restricted, it's access can't be more open then the inherited ones

    • Please register or sign in to reply
  • Please register or sign in to reply
    Loading