Draft: [skip ci] Add the ciStageJobsUpdated subscription

What does this MR do and why?

ciStageUpdated sends the job that changed, one update for each job transition. Its only consumer is the stage dropdown of the pipeline mini graph. A stage with hundreds of jobs then pushes hundreds of updates to an open dropdown, and the coalescer from !266624 (merged) cannot merge them, because each update is about a different job. In one MR pipeline, one stage got 668 triggers in a minute.

This MR adds ciStageJobsUpdated, which sends the stage instead:

  • Each update is the current state of all jobs in the stage. The client selects the jobs it needs, as in getPipelineStageJobs.
  • The subscription declares coalesce, with the default window of 500 milliseconds. A burst of job transitions in one stage becomes at most one update for each window, and the last one always arrives.
  • It reuses Subscriptions::Ci::Stages::StatusUpdated for the arguments, authorization and update, and returns CiStage, like ciStageStatusUpdated. ciStageStatusUpdated itself is not coalesced.

The trigger

  • On every job type. The trigger is on CommitStatus, not Ci::Build. Bridges and generic commit statuses now update their stage too. ciStageUpdated never covered them, so a trigger job in an open dropdown did not update.
  • No query. The trigger runs on every job transition, so it must not load the stage. It publishes the stage by its global ID through Gitlab::Graphql::Subscriptions::RecordReference, and the websocket nodes load the stage when they run the subscription. The topic argument has to be a GlobalID, not the URI::GID that Gitlab::GlobalId.build returns, because subscribers' topics hold its base64 form. The trigger spec checks both.

Rollout

The field is an experiment. The stack around it:

  1. This MR: the new field and trigger. Until the removal MR, both triggers publish, which doubles the Redis messages for stage updates.
  2. !266647 moves the mini graph to the new field.
  3. !266648 removes ciStageUpdated and its trigger, about 750 Redis messages each second on GitLab.com.

Merge the three close together, so the double publishing lasts as short as possible.

How to set up and validate locally

bin/rspec spec/graphql/subscriptions/ci/stages spec/graphql/graphql_triggers_spec.rb \
  spec/graphql/types/subscription_type_spec.rb spec/models/commit_status_spec.rb

Verified in GDK

With !266647 on top, the stage dropdown of a pipeline in gitlab-org/gitlab-test subscribed to ciStageJobsUpdated for its test stage (201 jobs). From a Rails runner, GraphqlTriggers.ci_stage_jobs_updated(job) was called 10 times within a few milliseconds:

graphql_subscription_coalescing Results sent to the browser
Off 11
On 3

The pipeline was running, so each run also included one real job transition, outside the burst. Each result carried the stage with its jobs, and the browser console had no errors.

Stage dropdown of the pipeline mini graph

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

References

Edited by Miguel Rincon

Merge request reports

Loading
Loading