Write web hook logs without a transaction (FF)

Hordur Freyr Yngvasonrequested to merge
fix/web-hook-log-insert-without-transaction into master

What does this MR do and why?

An attempt at buying pgbouncer connection pool capacity.

WebHooks::LogExecutionService writes each WebHookLog with create!, which wraps the INSERT in a transaction (BEGIN, INSERT, COMMIT). Between the INSERT and the COMMIT, Rails dumps every serialized column to YAML a second time in changes_applied. For large webhook payloads this keeps the connection idle in transaction (details below).

This MR adds WebHookLog.insert_log!(attributes). It validates the record, applies the same normalization as save (now grouped in prepare_for_storage, which the before_save callback also calls), sets created_at and updated_at from Ruby time as create! does, and writes the row with a single INSERT and no transaction.

  • insert is called with unique_by: %i[id created_at], because the partitioned table's primary key is (id, created_at) while the model declares primary_key = :id. The resulting ON CONFLICT DO NOTHING cannot trigger, since id comes from a sequence.
  • Timestamps are set in Ruby because insert would otherwise use the database clock.

The service uses insert_log! only when the new gitlab_com_derisk flag web_hook_log_insert_without_transaction is on. It is off by default. The actor is the project for project hooks, the group for group hooks, and Feature.current_request for system hooks. Both are built from the hook's IDs, so there is no extra query. This lets us roll out per project or group first. The flag is checked once per job and each hook always maps to the same actor, so mixing actor types is safe here. The docs use webhooks as the example (https://docs.gitlab.com/development/feature_flags/#mixing-actor-types):

In some situations it is safe to mix actor types if you know that it won't lead to inconsistent results. For example, a webhook can be associated with either a group or a project, and so a feature flag for a webhook might leverage this to roll out a feature for group and project webhooks using the same feature flag.

Why create! holds the transaction open longer than the INSERT

In Rails 7.2, create! runs changes_applied after the INSERT and before COMMIT. For each serialized column this calls Serialized#changed_in_place?, which YAML-dumps the value again. request_data holds the full payload, so this is CPU work done while the connection is idle in transaction.

Local GDK measurement, 3 runs each, 1.3 MB YAML payload:

Statements Time in transaction INSERT to COMMIT Total
create! BEGIN, INSERT, COMMIT about 110 ms about 96 ms about 390 ms
insert_log! single INSERT, no transaction none none about 280 ms

With a 123 KB payload, the INSERT-to-COMMIT gap was about 30 ms.

References

Screenshots or screen recordings

Not a UI change.

How to set up and validate locally

  1. In a Rails console, subscribe to sql.active_record and compare WebHookLog.create! (BEGIN, INSERT, COMMIT) with WebHookLog.insert_log! (a single INSERT) using the same attributes.
  2. Run Feature.enable(:web_hook_log_insert_without_transaction, project).
  3. Send a test push event from that project's Settings > Webhooks.
  4. Confirm only a single INSERT is issued, and the log appears under Recent events with credentials and emails redacted.

Database review

The queries below were captured locally by running WebHooks::LogExecutionService for a project hook with a push event payload, once with the flag off and once with it on. Query comments are removed.

The new query is a single-row INSERT, so there is no query plan. It still omits project_id, group_id, and organization_id, so trigger_web_hook_logs_daily_assign_sharding_keys fills them in as before.

The two inserts differ in three ways:

  • The transaction is gone.
  • RETURNING "id" is gone (returning: false), because the caller does not use the record.
  • ON CONFLICT ("id","created_at") DO NOTHING is added. This comes from unique_by, which must name the table's actual primary key. It cannot trigger, because id comes from a sequence.

Before (flag off): create!

BEGIN;

INSERT INTO "web_hook_logs_daily" ("web_hook_id", "trigger", "url", "request_headers", "request_data", "response_headers", "response_body", "response_status", "execution_duration", "updated_at", "created_at", "url_hash") VALUES (15, 'push_hooks', 'https://example.com/hook', '---
Content-Type: application/json
User-Agent: GitLab/19.5.0
X-Gitlab-Event: Push Hook
', '---
object_kind: push
event_name: push
ref: refs/heads/main
before: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
after: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
user_id: 1
user_name: Administrator
user_email: "[REDACTED]"
project_id: 1
project:
  id: 1
  name: Gitlab Smoke Tests
  web_url: http://gdk.test/toolbox/gitlab-smoke-tests
commits:
- id: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
  message: Update README
  author:
    name: Administrator
    email: "[REDACTED]"
total_commits_count: 1
', '---
Content-Type: text/plain
', 'ok', '200', 0.12, '2026-10-02 12:24:43.059400', '2026-10-02 12:24:43.059400', 'XXKqm5wHDYsaCPhC241sOC8pJo3NpfHlFW6bSIbGuG8=') RETURNING "id";

COMMIT;

After (flag on): insert_log!

INSERT INTO "web_hook_logs_daily" ("web_hook_id","trigger","url","request_headers","request_data","response_headers","response_body","response_status","execution_duration","updated_at","created_at","url_hash") VALUES (15, 'push_hooks', 'https://example.com/hook', '---
Content-Type: application/json
User-Agent: GitLab/19.5.0
X-Gitlab-Event: Push Hook
', '---
object_kind: push
event_name: push
ref: refs/heads/main
before: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
after: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
user_id: 1
user_name: Administrator
user_email: "[REDACTED]"
project_id: 1
project:
  id: 1
  name: Gitlab Smoke Tests
  web_url: http://gdk.test/toolbox/gitlab-smoke-tests
commits:
- id: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
  message: Update README
  author:
    name: Administrator
    email: "[REDACTED]"
total_commits_count: 1
', '---
Content-Type: text/plain
', 'ok', '200', 0.12, '2026-10-02 12:24:43.336004', '2026-10-02 12:24:43.336004', 'XXKqm5wHDYsaCPhC241sOC8pJo3NpfHlFW6bSIbGuG8=') ON CONFLICT ("id","created_at") DO NOTHING;

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Hordur Freyr Yngvason

Merge request reports