Add Get Web hook events API
What does this MR do and why?
Add get web hook events on group and project by REST API
Connected to #437188 (closed)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Create web hook event
- Request to API
curl --request GET \
--url http://gdk.test:3000/api/v4/<groups|projects>/<PARENT_ID>/hooks/<HOOK_ID>/events \
--header 'PRIVATE-TOKEN: <TOKEN>' \
WebHooks::WebHookLogsFinder
Example query plan from Query plans
Example of use:
hook = WebHook.first
user = User.first
WebHooks::WebHookLogsFinder.new(hook, user, hook.parent, { status: ['client_failure', 200, 500] }).execute
Example of raw SQL (8075115
is a webhook ID owned by gitlab-org/gitlab
):
SELECT "web_hook_logs".*
FROM "web_hook_logs"
WHERE "web_hook_logs"."web_hook_id" = 8075115
AND ( "web_hook_logs"."response_status" BETWEEN '400' AND '499'
OR "web_hook_logs"."response_status" = '200'
OR "web_hook_logs"."response_status" = '500' )
Example of query plan:
# In postgres.ai, first create the index being added in this MR:
exec CREATE INDEX web_hook_logs_on_web_hook_id_and_response_status ON ONLY web_hook_logs USING btree (web_hook_id, response_status);
https://console.postgres.ai/gitlab/gitlab-production-main/sessions/28697/commands/89415
Append (cost=0.57..46247.79 rows=12050 width=1382) (actual time=11.559..111781.689 rows=77803 loops=1)
Buffers: shared hit=27 read=78197 dirtied=1642
I/O Timings: read=110568.641 write=0.000
-> Index Scan using web_hook_logs_202402_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202402 web_hook_logs_1 (cost=0.57..37136.24 rows=9470 width=1389) (actual time=11.557..30403.693 rows=22217 loops=1)
Index Cond: (web_hook_logs_1.web_hook_id = 8075115)
Filter: ((((web_hook_logs_1.response_status)::text >= '400'::text) AND ((web_hook_logs_1.response_status)::text <= '499'::text)) OR ((web_hook_logs_1.response_status)::text = '200'::text) OR ((web_hook_logs_1.response_status)::text = '500'::text))
Rows Removed by Filter: 4
Buffers: shared hit=5 read=22318 dirtied=6
I/O Timings: read=30217.506 write=0.000
-> Index Scan using web_hook_logs_202403_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202403 web_hook_logs_2 (cost=0.57..2882.54 rows=741 width=1369) (actual time=8.838..21558.973 rows=15551 loops=1)
Index Cond: (web_hook_logs_2.web_hook_id = 8075115)
Filter: ((((web_hook_logs_2.response_status)::text >= '400'::text) AND ((web_hook_logs_2.response_status)::text <= '499'::text)) OR ((web_hook_logs_2.response_status)::text = '200'::text) OR ((web_hook_logs_2.response_status)::text = '500'::text))
Rows Removed by Filter: 1
Buffers: shared hit=10 read=15622 dirtied=15
I/O Timings: read=21425.277 write=0.000
-> Index Scan using web_hook_logs_202404_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202404 web_hook_logs_3 (cost=0.57..3141.61 rows=859 width=1368) (actual time=11.327..29921.386 rows=19370 loops=1)
Index Cond: (web_hook_logs_3.web_hook_id = 8075115)
Filter: ((((web_hook_logs_3.response_status)::text >= '400'::text) AND ((web_hook_logs_3.response_status)::text <= '499'::text)) OR ((web_hook_logs_3.response_status)::text = '200'::text) OR ((web_hook_logs_3.response_status)::text = '500'::text))
Rows Removed by Filter: 6
Buffers: shared hit=3 read=19486 dirtied=20
I/O Timings: read=29730.146 write=0.000
-> Index Scan using web_hook_logs_202405_web_hook_id_created_at_idx on gitlab_partitions_dynamic.web_hook_logs_202405 web_hook_logs_4 (cost=0.57..3027.14 rows=974 width=1347) (actual time=9.138..29854.263 rows=20665 loops=1)
Index Cond: (web_hook_logs_4.web_hook_id = 8075115)
Filter: ((((web_hook_logs_4.response_status)::text >= '400'::text) AND ((web_hook_logs_4.response_status)::text <= '499'::text)) OR ((web_hook_logs_4.response_status)::text = '200'::text) OR ((web_hook_logs_4.response_status)::text = '500'::text))
Rows Removed by Filter: 17
Buffers: shared hit=9 read=20771 dirtied=1601
I/O Timings: read=29195.713 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202406 web_hook_logs_5 (cost=0.00..0.00 rows=1 width=324) (actual time=0.015..0.015 rows=0 loops=1)
Filter: ((web_hook_logs_5.web_hook_id = 8075115) AND ((((web_hook_logs_5.response_status)::text >= '400'::text) AND ((web_hook_logs_5.response_status)::text <= '499'::text)) OR ((web_hook_logs_5.response_status)::text = '200'::text) OR ((web_hook_logs_5.response_status)::text = '500'::text)))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202407 web_hook_logs_6 (cost=0.00..0.00 rows=1 width=324) (actual time=0.008..0.008 rows=0 loops=1)
Filter: ((web_hook_logs_6.web_hook_id = 8075115) AND ((((web_hook_logs_6.response_status)::text >= '400'::text) AND ((web_hook_logs_6.response_status)::text <= '499'::text)) OR ((web_hook_logs_6.response_status)::text = '200'::text) OR ((web_hook_logs_6.response_status)::text = '500'::text)))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202408 web_hook_logs_7 (cost=0.00..0.00 rows=1 width=324) (actual time=0.007..0.007 rows=0 loops=1)
Filter: ((web_hook_logs_7.web_hook_id = 8075115) AND ((((web_hook_logs_7.response_status)::text >= '400'::text) AND ((web_hook_logs_7.response_status)::text <= '499'::text)) OR ((web_hook_logs_7.response_status)::text = '200'::text) OR ((web_hook_logs_7.response_status)::text = '500'::text)))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202409 web_hook_logs_8 (cost=0.00..0.00 rows=1 width=324) (actual time=0.006..0.006 rows=0 loops=1)
Filter: ((web_hook_logs_8.web_hook_id = 8075115) AND ((((web_hook_logs_8.response_status)::text >= '400'::text) AND ((web_hook_logs_8.response_status)::text <= '499'::text)) OR ((web_hook_logs_8.response_status)::text = '200'::text) OR ((web_hook_logs_8.response_status)::text = '500'::text)))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202410 web_hook_logs_9 (cost=0.00..0.00 rows=1 width=324) (actual time=0.007..0.007 rows=0 loops=1)
Filter: ((web_hook_logs_9.web_hook_id = 8075115) AND ((((web_hook_logs_9.response_status)::text >= '400'::text) AND ((web_hook_logs_9.response_status)::text <= '499'::text)) OR ((web_hook_logs_9.response_status)::text = '200'::text) OR ((web_hook_logs_9.response_status)::text = '500'::text)))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.web_hook_logs_202411 web_hook_logs_10 (cost=0.00..0.00 rows=1 width=324) (actual time=0.012..0.012 rows=0 loops=1)
Filter: ((web_hook_logs_10.web_hook_id = 8075115) AND ((((web_hook_logs_10.response_status)::text >= '400'::text) AND ((web_hook_logs_10.response_status)::text <= '499'::text)) OR ((web_hook_logs_10.response_status)::text = '200'::text) OR ((web_hook_logs_10.response_status)::text = '500'::text)))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
Merge request reports
Activity
Hey @lifez!
Thank you for your contribution to GitLab. Please refer to the contribution documentation for an overview of the process.
When you're ready for a first review, post
@gitlab-bot ready
. If you know a relevant reviewer(s) (for example, someone that was involved in a related issue), you can also assign them directly with@gitlab-bot ready @user1 @user2
.At any time, if you need help, feel free to post
@gitlab-bot help
or initiate a mentor session on Discord. Read more on how to get help.You can make AI-generated contributions to GitLab! If you use AI-generated content please check the box added to the top of your merge request description.
You can comment
@gitlab-bot label <label1> <label2>
to add labels to your MR. Please see the list of allowed labels in thelabel
command documentation.This message was generated automatically. You're welcome to improve it.
added Community contribution workflowin dev labels
assigned to @lifez
This merge request will be counted as part of the running Hackathon!
Find yourself in the Hackathon Leaderboard or check out the Hackathon page for more information!
This message was generated automatically. You're welcome to improve it.
added Hackathon label
This merge request will be counted as part of the running Hackathon!
Find yourself in the Hackathon Leaderboard or check out the Hackathon page for more information!
This message was generated automatically. You're welcome to improve it.
5 Warnings d52f2b12: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. d52f2b12: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. This merge request does not refer to an existing milestone. You've made some app changes, but didn't add any tests.
That's OK as long as you're refactoring existing code,
but please consider adding any of the maintenancepipelines, maintenancerefactor, maintenanceworkflow, documentation, QA labels.Labels missing: please ask a reviewer or maintainer to add databasereview pending, database, backend to this merge request. 2 Messages CHANGELOG missing: If this merge request needs a changelog entry, add the
Changelog
trailer to the commit message you want to add to the changelog.If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.
This merge request adds or changes files that require a review from the Database team. This merge request requires a database review. To make sure these changes are reviewed, take the following steps:
- Ensure the merge request has database and databasereview pending labels. If the merge request modifies database files, Danger will do this for you.
- Prepare your MR for database review according to the docs.
- Assign and mention the database reviewer suggested by Reviewer Roulette.
The following files require a review from the Database team:
app/finders/web_hooks/web_hook_logs_finder.rb
Reviewer roulette
Category Reviewer Maintainer backend @mcavoj
(UTC+2)
@rzwambag
(UTC+2)
database @jarka
(UTC+2)
@pshutsin
(UTC+2)
groupimport and integrate (backend) @jnutt
(UTC+1)
Maintainer review is optional for groupimport and integrate (backend) Please check reviewer's status!
Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangeradded linked-issue label
mentioned in issue #372826 (closed)
- Resolved by Luke Duncalfe
mentioned in merge request !150486 (merged)
requested review from @.luke
mentioned in issue gitlab-org/quality/triage-reports#17451 (closed)
changed milestone to %17.0
added Category:Webhooks groupimport and integrate labels
added devopsmanage sectioncore platform labels
added Leading Organization label
- A deleted user
added backend database databasereview pending labels
9 Warnings This MR changes code in ee/
, but its Changelog commit is missing theEE: true
trailer. Consider adding it to your Changelog commits.This merge request is quite big (1099 lines changed), please consider splitting it into multiple merge requests. 49e56a95: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 49e56a95: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 45282c1f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 45282c1f: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 3774af1d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. afcf29f3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. afcf29f3: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 1 Message This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/api/groups.md
(Link to current live version) -
doc/api/projects.md
(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Category Reviewer Maintainer backend @ddieulivol
(UTC+2)
@tianwenchen
(UTC+10)
database @carlad-gl
(UTC+2)
@jon_jenkins
(UTC-5)
groupimport and integrate (backend) @SamWord
(UTC-4)
Maintainer review is optional for groupimport and integrate (backend) Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger-
- Resolved by Phawin Khongkhasawan
- Resolved by Luke Duncalfe
- Resolved by Luke Duncalfe
@bhrai Are you able to give this Community contribution some early feedback?
-
!151048 (comment 1883364541) The API endpoint supports passing status codes, but also the strings
server_failure
andclient_failure
. We'd like to know how best to turnserver_failure
into a performant query. - !151048 (comment 1889296581) If an index is needed.
Thank you!
-
!151048 (comment 1883364541) The API endpoint supports passing status codes, but also the strings
requested review from @bhrai
@lifez Thank you, this direction looks good to me
. I've asked a database review for some early feedback. Besides whatever feedback comes from that early database review, we'd also want (you probably intend to do these, but I wanted to mention):- A changelog trailer
- Some spec coverage
Thank you
. Please@gitlab-bot ready @.luke
when you're readyremoved review request for @.luke
mentioned in issue gitlab-org/quality/triage-reports#17492 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17542 (closed)
removed review request for @bhrai
mentioned in issue gitlab-org/quality/triage-reports#17603 (closed)
added typefeature label
- Resolved by Phawin Khongkhasawan
@lifez, it seems we're waiting on an action from you for approximately two weeks.
- Do you still have capacity to work on this? If not, you might want to close this MR and/or ask someone to take over.
- Do you need help in getting it ready? At any time, you can:
- If you're actually ready for a review, you can post
@gitlab-bot ready
.
This message was generated automatically. You're welcome to improve it.
added automation:author-reminded label
added 5997 commits
-
d52f2b12...c83b27df - 5996 commits from branch
gitlab-org:master
- b665e800 - Add web_hook_logs_finder
-
d52f2b12...c83b27df - 5996 commits from branch
added pipelinetier-1 label
9 Warnings This MR changes code in ee/
, but its Changelog commit is missing theEE: true
trailer. Consider adding it to your Changelog commits.This merge request is quite big (1099 lines changed), please consider splitting it into multiple merge requests. 49e56a95: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 49e56a95: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 45282c1f: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 45282c1f: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 3774af1d: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. afcf29f3: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. afcf29f3: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 1 Message This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/api/groups.md
(Link to current live version) -
doc/api/projects.md
(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Category Reviewer Maintainer backend @ddieulivol
(UTC+2)
@mc_rocha
(UTC-4)
database @jarka
(UTC+2)
@jon_jenkins
(UTC-5)
groupimport and integrate (backend) @SamWord
(UTC-4)
Maintainer review is optional for groupimport and integrate (backend) Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
DangerEdited by Danger Bot-
added 527 commits
-
b665e800...f1ee4927 - 526 commits from branch
gitlab-org:master
- d755d4ad - Add web_hook_logs_finder
-
b665e800...f1ee4927 - 526 commits from branch
- Resolved by Luke Duncalfe
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Luke Duncalfe
@gitlab-bot ready @.luke
added workflowready for review label and removed workflowin dev label
requested review from @.luke
Hi
@lciutacu
! Please review this documentation merge request. This message was generated automatically. You're welcome to improve it.added documentation twtriaged labels
requested review from @lciutacu
@.luke
@lciutacu
, this Community contribution is ready for review.- Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
- If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.
This message was generated automatically. You're welcome to improve it.
removed automation:author-reminded label
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Lorena Ciutacu
Hi @lifez, I reviewed documentation and left some small suggestions for descriptions.
Edited by Lorena Ciutacu
added Technical Writing label
removed review request for @lciutacu
added pipeline:mr-approved label
- Resolved by Phawin Khongkhasawan
@lciutacu
, thanks for approving this merge request.This is the first time the merge request has been approved. To ensure we don't only run predictive pipelines, and we don't break
master
, please start a new pipeline before merging.Please wait for the pipeline to start before resolving this discussion and set auto-merge for the new pipeline. See merging a merge request for more details.
added pipelinetier-2 label and removed pipelinetier-1 label
changed milestone to %17.1
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
E2E Test Result Summary
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for 5c0d5a8dexpand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Verify | 44 | 0 | 3 | 0 | 47 | ✅ | | Create | 128 | 0 | 12 | 0 | 140 | ✅ | | Package | 19 | 0 | 12 | 0 | 31 | ✅ | | Data Stores | 31 | 0 | 0 | 0 | 31 | ✅ | | Govern | 69 | 0 | 0 | 0 | 69 | ✅ | | Plan | 67 | 0 | 2 | 0 | 69 | ✅ | | Monitor | 8 | 0 | 0 | 0 | 8 | ✅ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | | Manage | 0 | 0 | 1 | 0 | 1 | ➖ | | Release | 5 | 0 | 0 | 0 | 5 | ✅ | | Fulfillment | 1 | 0 | 0 | 0 | 1 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 374 | 0 | 30 | 0 | 404 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+
e2e-package-and-test:
test report for 5c0d5a8dexpand test summary
+-------------------------------------------------------------+ | suites summary | +--------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +--------+--------+--------+---------+-------+-------+--------+ | Manage | 30 | 0 | 15 | 2 | 45 | ✅ | | Create | 274 | 0 | 34 | 40 | 308 | ✅ | | Plan | 4 | 0 | 0 | 0 | 4 | ✅ | | Govern | 2 | 0 | 0 | 0 | 2 | ✅ | +--------+--------+--------+---------+-------+-------+--------+ | Total | 310 | 0 | 49 | 42 | 359 | ✅ | +--------+--------+--------+---------+-------+-------+--------+
e2e-test-on-cng:
test report for e69d8234expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Data Stores | 30 | 0 | 13 | 0 | 43 | ✅ | | Create | 138 | 0 | 16 | 0 | 154 | ✅ | | Package | 22 | 1 | 15 | 0 | 38 | ❌ | | Secure | 3 | 0 | 2 | 0 | 5 | ✅ | | Plan | 83 | 0 | 11 | 0 | 94 | ✅ | | Govern | 78 | 1 | 9 | 1 | 88 | ❌ | | Verify | 52 | 0 | 14 | 0 | 66 | ✅ | | Ai-powered | 0 | 0 | 1 | 0 | 1 | ➖ | | Monitor | 8 | 0 | 12 | 0 | 20 | ✅ | | Manage | 2 | 0 | 8 | 0 | 10 | ✅ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | | Configure | 0 | 0 | 3 | 0 | 3 | ➖ | | Fulfillment | 2 | 0 | 8 | 0 | 10 | ✅ | | ModelOps | 0 | 0 | 1 | 0 | 1 | ➖ | | Growth | 0 | 0 | 2 | 0 | 2 | ➖ | | Release | 5 | 0 | 1 | 0 | 6 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 425 | 2 | 116 | 1 | 543 | ❌ | +-------------+--------+--------+---------+-------+-------+--------+
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Luke Duncalfe
- Resolved by Phawin Khongkhasawan
Database migrations (on the main database)
6 Warnings 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
timing guidelines. Run time should not exceed 300000ms, but it was 2835566.06ms. Please consider
possible options to improve the query performance.CREATE INDEX CONCURRENTLY
"index_8c2627406d" ON "gitlab_partitions_dynamic"."web_hook_logs_202404" ("web_hook_id",
"response_status")20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
timing guidelines. Run time should not exceed 300000ms, but it was 2626449.66ms. Please consider
possible options to improve the query performance.CREATE INDEX CONCURRENTLY
"index_8782333851" ON "gitlab_partitions_dynamic"."web_hook_logs_202405" ("web_hook_id",
"response_status")20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
timing guidelines. Run time should not exceed 300000ms, but it was 2405871.16ms. Please consider
possible options to improve the query performance.CREATE INDEX CONCURRENTLY
"index_85b2fa997a" ON "gitlab_partitions_dynamic"."web_hook_logs_202403" ("web_hook_id",
"response_status")20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId had a query that exceeded
timing guidelines. Run time should not exceed 300000ms, but it was 2264890.57ms. Please consider
possible options to improve the query performance.CREATE INDEX CONCURRENTLY
"index_c2290bb536" ON "gitlab_partitions_dynamic"."web_hook_logs_202402" ("web_hook_id",
"response_status")20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId took 169.13min. Please add a
comment that mentions Release Managers (@gitlab-org/release/managers
) so they are informed.20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId may need a batched
background migration to comply with timing guidelines. It took 169.13min, but should not exceed 3.0minMigrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId Regular 10148.1 s +6.91 GiB Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 12 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 4 Migration: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId- Type: Regular
- Duration: 10148.1 s
- Database size change: +6.91 GiB
Calls Total Time Max Time Mean Time Rows Query 1 2835566.1 ms 2835566.1 ms 2835566.1 ms 0 CREATE INDEX CONCURRENTLY "index_8c2627406d" ON "gitlab_partitions_dynamic"."web_hook_logs_202404" ("web_hook_id", "response_status")
1 2626449.7 ms 2626449.7 ms 2626449.7 ms 0 CREATE INDEX CONCURRENTLY "index_8782333851" ON "gitlab_partitions_dynamic"."web_hook_logs_202405" ("web_hook_id", "response_status")
1 2405871.2 ms 2405871.2 ms 2405871.2 ms 0 CREATE INDEX CONCURRENTLY "index_85b2fa997a" ON "gitlab_partitions_dynamic"."web_hook_logs_202403" ("web_hook_id", "response_status")
1 2264890.6 ms 2264890.6 ms 2264890.6 ms 0 CREATE INDEX CONCURRENTLY "index_c2290bb536" ON "gitlab_partitions_dynamic"."web_hook_logs_202402" ("web_hook_id", "response_status")
1 53.4 ms 53.4 ms 53.4 ms 0 CREATE INDEX "web_hook_logs_on_web_hook_id_and_response_status" ON "web_hook_logs" ("web_hook_id", "response_status")
1 39.2 ms 39.2 ms 39.2 ms 0 CREATE INDEX CONCURRENTLY "index_b2c3e9b27b" ON "gitlab_partitions_dynamic"."web_hook_logs_202411" ("web_hook_id", "response_status")
1 31.7 ms 31.7 ms 31.7 ms 0 CREATE INDEX CONCURRENTLY "index_dd3617e333" ON "gitlab_partitions_dynamic"."web_hook_logs_202406" ("web_hook_id", "response_status")
1 19.9 ms 19.9 ms 19.9 ms 0 CREATE INDEX CONCURRENTLY "index_704bc80fa4" ON "gitlab_partitions_dynamic"."web_hook_logs_202408" ("web_hook_id", "response_status")
1 11.9 ms 11.9 ms 11.9 ms 0 CREATE INDEX CONCURRENTLY "index_9cfc33ea92" ON "gitlab_partitions_dynamic"."web_hook_logs_202407" ("web_hook_id", "response_status")
1 11.2 ms 11.2 ms 11.2 ms 0 CREATE INDEX CONCURRENTLY "index_e77f4400fb" ON "gitlab_partitions_dynamic"."web_hook_logs_202409" ("web_hook_id", "response_status")
1 10.8 ms 10.8 ms 10.8 ms 0 CREATE INDEX CONCURRENTLY "index_881be39fc6" ON "gitlab_partitions_dynamic"."web_hook_logs_202410" ("web_hook_id", "response_status")
1 3.5 ms 3.5 ms 3.5 ms 10 SELECT "postgres_partitions".*
FROM "postgres_partitions" WHERE "postgres_partitions"."parent_identifier" = $1
ORDER BY "postgres_partitions"."name" ASC1 0.9 ms 0.9 ms 0.9 ms 1 SELECT "feature_gates"."key", "feature_gates"."value" FROM "feature_gates" WHERE "feature_gates"."feature_key" = $1
1 0.3 ms 0.3 ms 0.3 ms 1 SELECT "postgres_partitioned_tables".*
FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
LIMIT $32 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Index creation timing exceededThese index creation queries might have caused your migration to exceed timing guidelines. Please examine the timing data on these queries and consider if the asynchronous index creation process is appropriate.
Histogram for AddWebhookLogIndexOnResponseStatusAndWebHookId
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 12 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 4 Other information
No other migrations pending on GitLab.com
Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-3330508-14135626-main
2024-05-31T04:03:50Z 2024-05-31T03:09:58Z 2024-05-31 19:02:57 +0000 database-testing-3330508-14135626-ci
2024-05-31T04:03:49Z 2024-05-31T03:45:22Z 2024-05-31 19:02:57 +0000 Database migrations (on the ci database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId Regular 16.6 s +80.00 KiB Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 16 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20240526130657 - AddWebhookLogIndexOnResponseStatusAndWebHookId
- Type: Regular
- Duration: 16.6 s
- Database size change: +80.00 KiB
Calls Total Time Max Time Mean Time Rows Query 1 15.0 ms 15.0 ms 15.0 ms 0 CREATE INDEX "web_hook_logs_on_web_hook_id_and_response_status" ON "web_hook_logs" ("web_hook_id", "response_status")
1 13.5 ms 13.5 ms 13.5 ms 0 CREATE INDEX CONCURRENTLY "index_c2290bb536" ON "gitlab_partitions_dynamic"."web_hook_logs_202402" ("web_hook_id", "response_status")
1 11.0 ms 11.0 ms 11.0 ms 0 CREATE INDEX CONCURRENTLY "index_881be39fc6" ON "gitlab_partitions_dynamic"."web_hook_logs_202410" ("web_hook_id", "response_status")
1 8.5 ms 8.5 ms 8.5 ms 0 CREATE INDEX CONCURRENTLY "index_9cfc33ea92" ON "gitlab_partitions_dynamic"."web_hook_logs_202407" ("web_hook_id", "response_status")
1 7.5 ms 7.5 ms 7.5 ms 0 CREATE INDEX CONCURRENTLY "index_704bc80fa4" ON "gitlab_partitions_dynamic"."web_hook_logs_202408" ("web_hook_id", "response_status")
1 7.4 ms 7.4 ms 7.4 ms 0 CREATE INDEX CONCURRENTLY "index_dd3617e333" ON "gitlab_partitions_dynamic"."web_hook_logs_202406" ("web_hook_id", "response_status")
1 7.1 ms 7.1 ms 7.1 ms 0 CREATE INDEX CONCURRENTLY "index_8c2627406d" ON "gitlab_partitions_dynamic"."web_hook_logs_202404" ("web_hook_id", "response_status")
1 6.6 ms 6.6 ms 6.6 ms 0 CREATE INDEX CONCURRENTLY "index_8782333851" ON "gitlab_partitions_dynamic"."web_hook_logs_202405" ("web_hook_id", "response_status")
1 6.1 ms 6.1 ms 6.1 ms 0 CREATE INDEX CONCURRENTLY "index_e77f4400fb" ON "gitlab_partitions_dynamic"."web_hook_logs_202409" ("web_hook_id", "response_status")
1 6.1 ms 6.1 ms 6.1 ms 0 CREATE INDEX CONCURRENTLY "index_b2c3e9b27b" ON "gitlab_partitions_dynamic"."web_hook_logs_202411" ("web_hook_id", "response_status")
1 6.0 ms 6.0 ms 6.0 ms 0 CREATE INDEX CONCURRENTLY "index_85b2fa997a" ON "gitlab_partitions_dynamic"."web_hook_logs_202403" ("web_hook_id", "response_status")
1 2.9 ms 2.9 ms 2.9 ms 10 SELECT "postgres_partitions".*
FROM "postgres_partitions" WHERE "postgres_partitions"."parent_identifier" = $1
ORDER BY "postgres_partitions"."name" ASC1 0.2 ms 0.2 ms 0.2 ms 1 SELECT "postgres_partitioned_tables".*
FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
LIMIT $32 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddWebhookLogIndexOnResponseStatusAndWebHookId
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 16 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
No other migrations pending on GitLab.com
Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-3330508-14135626-main
2024-05-31T04:03:50Z 2024-05-31T03:09:58Z 2024-05-31 19:02:57 +0000 database-testing-3330508-14135626-ci
2024-05-31T04:03:49Z 2024-05-31T03:45:22Z 2024-05-31 19:02:57 +0000
Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic
Edited by ****- Resolved by Luke Duncalfe
@jarka Could you please give this Community contribution its first database review? I left an example explain in the MR description, and I suspect we need to optimise the query a bit further. I left a thought here !151048 (comment 1930094794), and very interested to know how we could improve it
requested review from @jarka
removed review request for @.luke
added database-testing-automation label
reset approvals from @lciutacu by pushing to the branch
mentioned in issue gitlab-org/quality/triage-reports#18061 (closed)
removed review request for @jarka
added pipelinetier-1 label and removed pipelinetier-2 label
Hi @jarka
We noticed this MR is marked as workflowready for review but no reviewer is assigned. workflowin dev has automatically been applied to this MR based on the likelihood the review is finished. If additional reviews are still required, please assign a reviewer and reapply workflowready for review.
@lifez you may also request a review by commenting
@gitlab-bot ready
. You can also assign reviewers directly using@gitlab-bot ready @user1 @user2
if you know the relevant reviewer(s), such as those who were involved in a related issue.This message was generated automatically. You're welcome to improve it.
added workflowin dev label and removed workflowready for review label
- Resolved by Luke Duncalfe
mentioned in issue #465539 (closed)
@gitlab-bot ready @.luke
added workflowready for review label and removed workflowin dev label
requested review from @.luke
- Resolved by Luke Duncalfe
@.luke
, this Community contribution is ready for review.- Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
- If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.
This message was generated automatically. You're welcome to improve it.
mentioned in commit gitlab-community/gitlab@12b791d3
mentioned in commit gitlab-community/gitlab@42c1815f
mentioned in commit gitlab-community/gitlab@3c793918
mentioned in commit gitlab-community/gitlab@af21738f
added 1905 commits
-
3c793918...8fd58c1c - 1898 commits from branch
gitlab-org:master
- 796eb442 - Add web_hook_logs_finder
- 98b7c632 - Add document
- b54bcdcf - Add web_hook_logs_finder_spec
- ca2a45a4 - Apply 3 suggestion(s) to 2 file(s)
- cd030fea - Apply 4 suggestion(s) to 2 file(s)
- 3bafce42 - Apply suggestions
- af21738f - Add Webhook log recent API
Toggle commit list-
3c793918...8fd58c1c - 1898 commits from branch
- Resolved by Phawin Khongkhasawan
mentioned in issue gitlab-org/quality/triage-reports#18161 (closed)
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
@lifez Thank you very much for all of your efforts here
! I appreciate all of your time here very much. I've left some new suggestions! Please@gitlab-bot ready @.luke
when ready!removed review request for @.luke
Hi @.luke
We noticed this MR is marked as workflowready for review but no reviewer is assigned. workflowin dev has automatically been applied to this MR based on the likelihood the review is finished. If additional reviews are still required, please assign a reviewer and reapply workflowready for review.
@lifez you may also request a review by commenting
@gitlab-bot ready
. You can also assign reviewers directly using@gitlab-bot ready @user1 @user2
if you know the relevant reviewer(s), such as those who were involved in a related issue.This message was generated automatically. You're welcome to improve it.
added workflowin dev label and removed workflowready for review label
- Resolved by Luke Duncalfe
- Resolved by Luke Duncalfe
mentioned in commit gitlab-community/gitlab@5efcd699
mentioned in commit gitlab-community/gitlab@84abb627
added 1741 commits
-
5efcd699...46f4b7dc - 1734 commits from branch
gitlab-org:master
- f403116c - Add web_hook_logs_finder
- 8c174a3a - Add document
- 168f29f9 - Add web_hook_logs_finder_spec
- 6b945c12 - Apply 3 suggestion(s) to 2 file(s)
- 8197563e - Apply 4 suggestion(s) to 2 file(s)
- c769a3f2 - Apply suggestions
- 84abb627 - Add Webhook log recent API
Toggle commit list-
5efcd699...46f4b7dc - 1734 commits from branch
- Resolved by Luke Duncalfe
@gitlab-bot ready @.luke
added workflowready for review label and removed workflowin dev label
requested review from @.luke
@.luke
, this Community contribution is ready for review.- Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
- If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.
This message was generated automatically. You're welcome to improve it.
mentioned in commit gitlab-community/gitlab@f5854735
mentioned in issue gitlab-org/quality/triage-reports#18244 (closed)
- Resolved by Phawin Khongkhasawan
- Resolved by Luke Duncalfe
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
removed review request for @.luke
added workflowin dev label and removed workflowready for review label
changed milestone to %17.2
requested review from @jarka
mentioned in commit gitlab-community/gitlab@a3236b1e
- Resolved by Luke Duncalfe
@gitlab-bot ready @.luke
added workflowready for review label and removed workflowin dev label
requested review from @.luke
@jarka
@.luke
, this Community contribution is ready for review.- Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
- If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.
This message was generated automatically. You're welcome to improve it.
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
requested review from @eread
- Resolved by James Nutt
added pipelinetier-2 label and removed pipelinetier-1 label
- Resolved by Luke Duncalfe
Before you set this MR to auto-merge
This merge request will progress on pipeline tiers until it reaches the last tier: pipelinetier-3.
Before you resolve this discussion, please check the following:
- You are the last maintainer of this merge request
- The latest pipeline for this merge request is pipelinetier-3 (You can find which tier it is in the pipeline name)
- This pipeline is recent enough (created in the last 8 hours)
If all the criteria above apply, please resolve this discussion and the set auto-merge for this merge request.
See pipeline tiers and merging a merge request for more details.
removed review request for @eread
- Resolved by Phawin Khongkhasawan
@lifez Thank you! I've left one suggestion, but this looks good to me so I'll approve now! There will be some changes to the finder, but I think the discussion is heading in the right direction, so I'll pass to a backend maintainer now.
@jnutt Could you please give this Community contribution MR its backend maintainer review?
reset approvals from @.luke by pushing to the branch
mentioned in issue gitlab-org/quality/triage-reports#18354 (closed)
reset approvals from @eread by pushing to the branch
- Resolved by James Nutt
- Resolved by Pavel Shutsin
added databasereviewed label and removed databasereview pending label
mentioned in issue gitlab-com/www-gitlab-com#32017 (closed)
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Luke Duncalfe
requested review from @pshutsin
added databaseapproved label and removed databasereviewed label
removed review request for @pshutsin
added pipelinetier-3 label and removed pipelinetier-2 label
added 2623 commits
-
aee13dbb...a75c9517 - 2612 commits from branch
gitlab-org:master
- 3b332b56 - 1 earlier commit
- 1607390f - Add document
- 5007866c - Add web_hook_logs_finder_spec
- e6e9750d - Apply 3 suggestion(s) to 2 file(s)
- fec7cd39 - Apply 4 suggestion(s) to 2 file(s)
- ff9e5cb9 - Apply suggestions
- cd7c4ea7 - Add Webhook log recent API
- c217df84 - Apply 2 suggestion(s) to 2 file(s)
- d21ccf98 - Apply 1 suggestion(s) to 1 file(s)
- 47e5e0c4 - Apply 1 suggestion(s) to 1 file(s)
- 52b42e1d - Apply 1 suggestion(s) to 1 file(s)
Toggle commit list-
aee13dbb...a75c9517 - 2612 commits from branch
mentioned in issue gitlab-org/quality/triage-reports#18433 (closed)
@gitlab-bot ready @.luke
requested review from @.luke
- Resolved by Phawin Khongkhasawan
- Resolved by Phawin Khongkhasawan
- Resolved by Luke Duncalfe
- Resolved by Luke Duncalfe
mentioned in issue gitlab-org/quality/triage-reports#18629 (closed)
mentioned in merge request gitlab-com/www-gitlab-com!134633 (merged)
mentioned in merge request !151130 (merged)
- Resolved by Luke Duncalfe
@lifez Apologies for the delay
thank you so much on all of your hard work here, I've really appreciated your collaborative attitude and all of the thoughtful conversations .We've lost the database approval that is required by our Codeowners rules which prevents me from merging yet. @pshutsin or @jarka could you please re-apply your approval
removed review request for @jarka
There's an undercoverage error https://gitlab.com/gitlab-org/gitlab/-/jobs/7302009284 since the pipelinetier-3 label was applied. I'm going to apply the pipeline:skip-undercoverage label because I worry we could get stuck in a cycle of waiting for approvals, pushing changes, and waiting for approvals again.
added pipeline:skip-undercoverage label
removed review request for @pshutsin
- Resolved by Luke Duncalfe
Set to merge when the pipeline succeeds
!
mentioned in incident gitlab-org/quality/engineering-productivity/approved-mr-pipeline-incidents#795 (closed)
- Resolved by Luke Duncalfe
requested review from @pshutsin
removed review request for @pshutsin
mentioned in issue gitlab-org/quality/triage-reports#18716 (closed)
added 2586 commits
-
c237c027...c82bf583 - 2574 commits from branch
gitlab-org:master
- c82bf583...3774af1d - 2 earlier commits
- 45282c1f - Add web_hook_logs_finder_spec
- b1ffee2d - Apply 3 suggestion(s) to 2 file(s)
- d222d12f - Apply 4 suggestion(s) to 2 file(s)
- 49e56a95 - Apply suggestions
- 85fcfa7d - Add Webhook log recent API
- 9acc6a9b - Apply 2 suggestion(s) to 2 file(s)
- d1488742 - Apply 1 suggestion(s) to 1 file(s)
- ba37d1cc - Apply 1 suggestion(s) to 1 file(s)
- 62764898 - Apply 1 suggestion(s) to 1 file(s)
- 58d1ea36 - Apply 2 suggestion(s) to 2 file(s)
Toggle commit list-
c237c027...c82bf583 - 2574 commits from branch
- Resolved by Luke Duncalfe
@gitlab-bot ready @pshutsin
requested review from @pshutsin
changed milestone to %17.3
removed review request for @pshutsin
@gitlab-bot ready @pshutsin
requested review from @pshutsin
Let's merge! Thank you @lifez!@lifez, how was your code review experience with this merge request? Please tell us how we can continue to iterate and improve:
- React with a
or a on this comment to describe your experience. - Create a new comment starting with
@gitlab-bot feedback
below, and leave any additional feedback you have for us in the comment.
Subscribe to the GitLab Community Newsletter for contributor-focused content and opportunities to level up.
Thanks for your help!
This message was generated automatically. You're welcome to improve it.
- React with a
Hello @lifez
The database team is looking for ways to improve the database review process and we would love your help!
If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:
@gitlab-org/database-team
And someone will be by shortly!
Thanks for your help!
This message was generated automatically. You're welcome to improve it.
mentioned in incident gitlab-org/quality/engineering-productivity/master-broken-incidents#7446 (closed)
30187 30187 30188 30188 CREATE UNIQUE INDEX vulnerability_occurrence_pipelines_on_unique_keys ON vulnerability_occurrence_pipelines USING btree (occurrence_id, pipeline_id); 30189 30189 30190 CREATE INDEX web_hook_logs_on_web_hook_id_and_response_status ON ONLY web_hook_logs USING btree (web_hook_id, response_status); 30191 @pshutsin This should not be included, as the index is not created yet. It should be added when the index is created synchronously.
@.luke Should we merge all this before the index is in place?
@krasio Good catch, perhaps not. Do we need to remove this line?
Error: db/structure.sql was changed, and no migrations were added: diff --git a/db/structure.sql b/db/structure.sql index 403ff298643..8ac45e96e46 100644 --- a/db/structure.sql +++ b/db/structure.sql @@ -30181,8 +30181,6 @@ CREATE UNIQUE INDEX virtual_registries_pkgs_maven_registries_on_unique_group_ids
Got this error while commit. Should I skip the lefthook?
@lifez Yes I think so
Here's a plan for the filter by status query, when this index is present. We can see it's not being used.
gitlabhq_dblab=# explain (analyze , buffers ) SELECT gitlabhq_dblab-# "web_hook_logs".* gitlabhq_dblab-# FROM gitlabhq_dblab-# "web_hook_logs" gitlabhq_dblab-# WHERE gitlabhq_dblab-# "web_hook_logs"."web_hook_id" = 25401284 gitlabhq_dblab-# AND "web_hook_logs"."response_status" = '400' gitlabhq_dblab-# AND "web_hook_logs"."created_at" BETWEEN '2024-07-12 00:00:00' gitlabhq_dblab-# AND '2024-07-19 00:42:17.342483' gitlabhq_dblab-# ORDER BY gitlabhq_dblab-# "web_hook_logs"."created_at" DESC gitlabhq_dblab-# LIMIT gitlabhq_dblab-# 20 OFFSET 0; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Limit (cost=0.57..15367.85 rows=20 width=1350) (actual time=131414.673..131414.686 rows=0 loops=1) Buffers: shared hit=38860 read=283066 I/O Timings: read=128744.419 -> Index Scan Backward using web_hook_logs_202407_created_at_web_hook_id_idx on web_hook_logs_202407 web_hook_logs (cost=0.57..3937865.87 rows=5125 width=1350) (actual time=131414.670..131414.671 rows=0 loops=1) Index Cond: ((created_at >= '2024-07-12 00:00:00'::timestamp without time zone) AND (created_at <= '2024-07-19 00:42:17.342483'::timestamp without time zone) AND (web_hook_id = 25401284)) Filter: ((response_status)::text = '400'::text) Rows Removed by Filter: 45961 Buffers: shared hit=38860 read=283066 I/O Timings: read=128744.419 Planning: Buffers: shared hit=14 read=5 I/O Timings: read=8.785 Planning Time: 10.862 ms Execution Time: 131414.766 ms (14 rows) Time: 131742.847 ms (02:11.743) gitlabhq_dblab=# \d web_hook_logs_on_web_hook_id_and_response_status Partitioned index "public.web_hook_logs_on_web_hook_id_and_response_status" Column | Type | Key? | Definition -----------------+-------------------+------+----------------- web_hook_id | integer | yes | web_hook_id response_status | character varying | yes | response_status btree, for table "public.web_hook_logs" Number of partitions: 1 (Use \d+ to list them.)
@krasio Oh that's interesting. The query took 2 minutes to complete? So we're missing an index still, but we haven't managed to define one that PG will use, is that right? Would the correct index be on
(web_hook_id, created_at, response_status)
or perhaps(created_at, web_hook_id, response_status)
I'm not sure which order is best.Edited by Luke DuncalfeIndex on
(created_at, web_hook_id, response_status)
helps, but it's still not ideal.Examples
gitlabhq_dblab=# explain (analyze, buffers) SELECT "web_hook_logs".* FROM "web_hook_logs" WHERE "web_hook_logs"."web_hook_id" = 25401284 AND "web_hook_logs"."response_status" = '400' AND "web_hook_logs"."created_at" BETWEEN '2024-07-12 00:00:00' AND '2024-07-19 00:42:17.342483' ORDER BY "web_hook_logs"."created_at" DESC LIMIT 20 OFFSET 0; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Limit (cost=0.57..5005.63 rows=20 width=1350) (actual time=4973.807..4973.809 rows=0 loops=1) Buffers: shared hit=4 read=236947 I/O Timings: read=3029.496 -> Index Scan Backward using index_1_202407 on web_hook_logs_202407 web_hook_logs (cost=0.57..1282547.15 rows=5125 width=1350) (actual time=4973.803..4973.803 rows=0 loops=1) Index Cond: ((created_at >= '2024-07-12 00:00:00'::timestamp without time zone) AND (created_at <= '2024-07-19 00:42:17.342483'::timestamp without time zone) AND (web_hook_id = 25401284) AND ((response_status)::text = '400'::text)) Buffers: shared hit=4 read=236947 I/O Timings: read=3029.496 Planning: Buffers: shared read=5 I/O Timings: read=0.259 Planning Time: 0.592 ms Execution Time: 4973.880 ms (12 rows) Time: 5282.178 ms (00:05.282) gitlabhq_dblab=# explain (analyze, buffers) SELECT "web_hook_logs".* FROM "web_hook_logs" WHERE "web_hook_logs"."web_hook_id" = 25401284 AND "web_hook_logs"."response_status" BETWEEN '500' AND '599' AND "web_hook_logs"."created_at" BETWEEN '2024-07-12 00:00:00' AND '2024-07-19 00:42:17.342483' ORDER BY "web_hook_logs"."created_at" DESC LIMIT 20 OFFSET 0; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Limit (cost=0.57..6838.58 rows=20 width=1350) (actual time=4532.437..4532.439 rows=0 loops=1) Buffers: shared hit=4 read=236947 I/O Timings: read=2581.960 -> Index Scan Backward using index_1_202407 on web_hook_logs_202407 web_hook_logs (cost=0.57..1435297.89 rows=4198 width=1350) (actual time=4532.434..4532.435 rows=0 loops=1) Index Cond: ((created_at >= '2024-07-12 00:00:00'::timestamp without time zone) AND (created_at <= '2024-07-19 00:42:17.342483'::timestamp without time zone) AND (web_hook_id = 25401284) AND ((response_status)::text >= '500'::text) AND ((response_status)::text <= '599'::text)) Buffers: shared hit=4 read=236947 I/O Timings: read=2581.960 Planning: Buffers: shared read=5 I/O Timings: read=0.191 Planning Time: 0.525 ms Execution Time: 4532.495 ms (12 rows) Time: 4872.137 ms (00:04.872)
Edited by Krasimir Angelov@krasio Are these queries using the pagination limit of
20
?@.luke Yes. Sorry, the SQL was there but formatting was off so it was not visible. Fixed.
BTW
web_hook_logs
is one of the table we should be careful to add index to - https://gitlab.com/gitlab-org/gitlab/-/blob/84b753603028a2de854b50c3afbeaaf32e9c9b41/rubocop/cop/migration/prevent_index_creation.rb#L27. It was missed because the cop does not check forprepare_partitioned_async_index
.
added workflowstaging-canary label and removed workflowready for review label
added workflowcanary label and removed workflowstaging-canary label
mentioned in merge request !159862 (merged)
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
- Resolved by Luke Duncalfe
mentioned in merge request !160184 (merged)
added releasedcandidate label
mentioned in commit b8130f74
mentioned in merge request !162255 (merged)
mentioned in commit dca0e967
mentioned in commit sac-valley-tech/forks/gitlab@9ad8f931
added releasedpublished label and removed releasedcandidate label
Sorry to bother you guys. I saw this PR in the 17.3 release blog post and noticed that this community contribution is not available for 'free' users , neither SaaS or gitlab.com. That surprised me. Is this something new, is that an issue in the changelog or was it always possible to find community contributions behind the payed versions of gitlab?
@lexxxel It working fine on the project scope.
curl --request GET \ --url https://gitlab.com/api/v4/projects/60564468/hooks/43037580/events \ --header 'PRIVATE-TOKEN: <PROJECT_ACCESS_TOKEN>' \
On the other hand, group webhook are only available to Premium and Ultimate users.
Btw, Should we consider further clarifying or separating the release post between Project and Group hooks? @m_frankiewicz
@lifez that makes sense. If the overlying feature is already restricted, it's access can't be more open then the inherited ones