Follow-up to https://gitlab.com/gitlab-org/gitlab/-/issues/608670, the corrective action for https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29434 (INC-12449: a GET endpoint writing to the database on every request took down the primary under a traffic burst). The `detect_writes_on_get` analyzer (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/250734) now logs every data-modifying query served during a GET or HEAD request in production, currently at 15% of traffic. `Projects::NotesController#index` is in the top ten GET endpoints writing to the database. | Endpoint | Write | Where it comes from | |---|---|---| | `Projects::NotesController#index` (top 10) | `UPDATE resource_label_events SET reference` | `LabelNote#update_outdated_reference` (`app/models/label_note.rb:53`) calls `ResourceLabelEvent#refresh_invalid_reference`, which fills `reference` for events never rendered and does `save if changed?` (`app/models/resource_label_event.rb:49-56`) | The boards controllers write on GET too (`Projects::BoardsController#index`/`#show` and the group equivalents, all allowlisted today). That is out of scope for this issue and tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/628465. Logs: https://log.gprd.gitlab.net/app/r/s/CwmW7 (filter by `json.endpoint_id`; each entry carries the SQL, the tables written and the stack trace). Overview of all endpoints: https://log.gprd.gitlab.net/app/r/s/h15ms. Top 10 with stack traces: https://gitlab.com/gitlab-org/gitlab/-/issues/608670#note_3789467415. ## Proposal Keep the stored `reference` column and write it at every path that creates or repoints a `resource_label_events` row, instead of deriving it at render time and dropping the column. A cross-project or cross-group label reference is expensive to resolve at render. The query count backs that up: a warm read of a GDK issue with 30 label events is 16 queries with the stored value, 20 if derived. The stored column is the cheaper read. The original proposal ruled this out because five paths left `reference` NULL on creation. Each is now covered: - Normal label add and remove: `ResourceEvents::ChangeLabelsService` computes `reference` into the bulk-insert hash. - Issue move and clone: `Gitlab::Issuable::Clone::CopyResourceEventsService` recomputes the reference at copy time against the new parent. - Project import/export and GitHub import: a `before_validation :ensure_reference, on: :create` callback on `ResourceLabelEvent` sets it for single-row creates. - Label promotion: `Labels::PromoteService` sets `reference` when it repoints events at the group label. This also fixes a live bug: promoted labels used to leave events pointing at a destroyed project label forever. - Label deletion: the events are cascade-deleted instead of left pointing at a dead label. Rows written before this shipped get a batched background migration backfill. The read path then drops the `save if changed?` call entirely. MR 7 adds the zero-write spec this issue asked for: a regression test in `spec/controllers/projects/notes_controller_spec.rb` asserting no write to `resource_label_events` when `Projects::NotesController#index` handles a GET. ## Merge requests Seven MRs, all Draft and all milestone %19.5, each targeting the previous branch. Merge in this order. The numbers match the branch names, so there is no MR 2: !255083 was folded into !255091 and closed on 2026-09-13. | # | MR | Title | Required for INC-12449? | |---|---|---|---| | 1 | !255082 | Render group level work item label notes against the group | No. Rides along; fixes a separate bug where group-level work items passed `work_item.project`, nil, as the note parent. | | 3 | !255091 | Store the label event reference when the event is written | Yes. The write side of the fix: computes and persists `reference` at every creation and repoint path listed above. | | 4 | !255092 | Delete label events when their label is deleted | No. Rides along; cascade-deletes events instead of nullifying `label_id`. Drops the "removed 1 deleted label" timeline line. | | 4b | !255238 | Record deleted label events in the label audit event | No. Goes with MR 4. Reports how many timeline entries the deletion removed in the existing EE `label_deleted` audit event. | | 5 | !255093 | Link a project label in a group context to its project | Needed before MR 6. Fixes a Banzai URL bug that would otherwise make backfilled rows render dead links. | | 6 | !255096 | Backfill the reference on existing label events | Yes. Batched background migration fills `reference` for rows written before MR 3. | | 7 | !255097 | Stop writing the label event reference on a read request | Yes. The actual incident fix: removes the `save` from the read path. | The spine for the incident is MR 3 -> MR 6 -> MR 7 (MR 5 is a prerequisite of MR 6). MRs 1, 4 and 4b ride along at the author's choice and are not required to close the incident action. ## Deferred to 19.6 - The migration that finalizes the backfill background migration cannot be written yet. `prevent_early_finalization!` needs the queueing milestone to be at or before the last required upgrade stop in `config/upgrade_path.yml`. The queueing migration is %19.5, and `Gitlab::Utils::UpgradePath#last_required_stop` picks stops strictly below the running version, so at app 19.5 the last stop is still 19.2. The finalize migration is writable from 19.6. - Once the backfill has finished on gitlab.com, remove the derive-on-read path entirely: `LabelNote#update_outdated_reference`, `ResourceLabelEvent#outdated_reference?`, and `#refresh_invalid_reference`. ## Changes from the original description The original text is preserved at the bottom of this description. - Split the boards work (default board creation, recent-visits upsert) into https://gitlab.com/gitlab-org/gitlab/-/work_items/628465, since it is a separate write path with a separate fix. This issue now covers only `Projects::NotesController#index`. - The original closing sentence said to remove "these endpoints" from `ALLOWED_ENDPOINTS`. Only the boards endpoints are in that list; `Projects::NotesController#index` never was. The allowlist cleanup belongs to the boards follow-up. - The original closing sentence said the analyzer raises in dev and test. `PreventWritesOnGet` only logs a warning; `PreventCrossDatabaseModification` is the one that raises. This issue needs its own explicit zero-write spec instead of relying on an automatic guard. - 2026-09-11: dropped the derive-at-render design (below) for the write-at-entry-points design above. A cross-project or cross-group label reference is expensive to resolve at render, and the stored column is the cheaper read (16 queries warm versus 20 derived). The work item author decided this during implementation. - 2026-09-13: a five-agent review found three code blockers (a writer/renderer mismatch for group parents, a parity test that never ran, and a wrong reference on events promoted across groups). All three are fixed and the whole chain was rewritten and force-pushed. MR 2 was folded into MR 3 and closed. MR 4b was added so a label deletion records how many timeline entries it destroyed. All migrations moved from 19.4 to 19.5. <details> <summary>Derive-at-render proposal (superseded 2026-09-11)</summary> ## Proposal Computing `reference` at write time does not work. Nothing sets it at creation today, on any path. Five paths create rows with `reference` NULL: normal label changes, issue move and clone (which deliberately excludes the column so it gets recomputed against the new parent), project import/export, GitHub import, and label deletion. Filling it only at creation would leave the move, clone and import rows NULL forever, and a NULL `reference` is not `present?`, so the note would keep rendering "added 1 deleted label" instead of the real label. The approach instead: stop persisting `reference` at all, derive it at render time from the already-loaded label, then drop the column. No backfill is needed, because the value stops being stored. This also fixes a live bug: `Labels::PromoteService` remaps `label_id` to the new group label and destroys the old project label without touching `reference`, and nothing ever repairs that row, so the note renders a bare `~<id>` forever. Deriving at render time costs three batching fixes, without which it is a large performance regression. Measured on a GDK issue with 30 label events: naive derivation runs 177 queries per render against today's 16, and with the batching fixes it is 20 queries and zero writes. When fixed, add an explicit spec asserting zero writes when `Projects::NotesController#index` is called, since `Gitlab::Database::QueryAnalyzers::PreventWritesOnGet` only logs a warning today and has no automatic guard in dev or test (`PreventCrossDatabaseModification` is the analyzer that raises in dev and test; this one does not). Giving `PreventWritesOnGet` a dev/test raise could be worth its own issue. The allowlist cleanup for the four boards endpoints belongs to the boards follow-up, not to this issue, since `Projects::NotesController#index` was never in `ALLOWED_ENDPOINTS`. ## Phases 1. Render group level work item label notes against the group. `work_item_synthetic_system_note` passed `work_item.project` as the note's parent, which is nil for group level work items, so the note rendered raw text like "added ~52 label" instead of a label chip. Switches to `work_item.resource_parent`; independent of the rest of the stack. 1. Make `ResourceLabelEvent#label_reference` public and add optional `parent:` and `local_label_ids:` arguments so a batch of events can share one label lookup, returning an empty string for a deleted label. No caller passes the new arguments yet, so behavior is unchanged; also passes the container object to `LabelsFinder` instead of its id, and makes a nil parent return false instead of an unscoped query. 1. Derive the reference at render time behind the `derive_label_event_reference` feature flag (type `gitlab_com_derisk`, default off), with the stored column still read when the flag is off. The builder service resolves reachable labels once per page and preloads label container routes to avoid a query and a route lookup per event. 1. Remove the flag and the write, deleting `LabelNote#update_outdated_reference`, `ResourceLabelEvent#outdated_reference?`, and `ResourceLabelEvent#refresh_invalid_reference`. This is the MR where the full test suite runs against the new render path. 1. Add `ignore_column :reference`. 1. Post-deployment migration to drop the column, run under `with_lock_retries` since `resource_label_events` is on the high traffic table list. 1. Remove the `ignore_column` entry, the clone service's except-list entry for `reference` and `reference_html`, and the import/export `excluded_attributes` entry. MRs 1 through 4 are sufficient to close the incident action; MRs 5 through 7 are the standard three-release column removal and must each land in a separate milestone. Two user visible bugs get fixed as a side effect and should be called out in the MR descriptions: promoting a label to its group leaves every event pointing at a destroyed project label, so the note renders a dead reference forever, and the group level work item parent bug fixed in MR 1. </details> <details> <summary>Original version</summary> Follow-up to https://gitlab.com/gitlab-org/gitlab/-/issues/608670, the corrective action for https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29434 (INC-12449: a GET endpoint writing to the database on every request took down the primary under a traffic burst). The `detect_writes_on_get` analyzer (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/250734) now logs every data-modifying query served during a GET or HEAD request in production, currently at 15% of traffic. `Projects::NotesController#index` is in the top ten GET endpoints writing to the database, and the boards controllers, allowlisted in the analyzer today, write on GET too. | Endpoint | Write | Where it comes from | |---|---|---| | `Projects::NotesController#index` (top 10) | `UPDATE resource_label_events SET reference` | `LabelNote#update_outdated_reference` (`app/models/label_note.rb:53`) calls `ResourceLabelEvent#refresh_invalid_reference`, which fills `reference` for events never rendered and does `save if changed?` (`app/models/resource_label_event.rb:49-56`) | | `Projects::BoardsController#index`, `Groups::BoardsController#index` (allowlisted) | `INSERT INTO boards` (+ lists) | `BoardsActions#index` creates a default board when the parent has none (`app/controllers/concerns/boards_actions.rb:16`) | | `Projects::BoardsController#show`, `Groups::BoardsController#show` (allowlisted) | recent visits upsert | `BoardsActions#show` records the visit (`app/controllers/concerns/boards_actions.rb:22`) | Logs: https://log.gprd.gitlab.net/app/r/s/CwmW7 (filter by `json.endpoint_id`; each entry carries the SQL, the tables written and the stack trace). Overview of all endpoints: https://log.gprd.gitlab.net/app/r/s/h15ms. Top 10 with stack traces: https://gitlab.com/gitlab-org/gitlab/-/issues/608670#note_3789467415. ## Proposal - Compute `reference` at write time (when the event is created, and when a label is deleted) and drop the lazy `save` from the read path. If old rows need a backfill, do it in a batched background migration rather than on first render. - Create the default board on the write path (project or group creation) or in a worker triggered by the first visit, and record recent visits in a worker or with a throttle, as `Users::ActivityService` does. When fixed, remove these endpoints from `ALLOWED_ENDPOINTS` in `lib/gitlab/database/query_analyzers/prevent_writes_on_get.rb`. In dev and test the analyzer raises, so a spec covering the GET will fail if the write comes back. </details>