Send the mobile push summaries when working hours begin

Marcel van Remmerdenrequested to merge
mvanremmerden-push-working-hours-summaries into master

What does this MR do and why?

When a user's working hours begin, they get one push, "You have new to-dos", with the pending to-do count as badge, for the to-dos held while their pushes were paused.

  • Notifications::MobilePush::WorkingHoursResumeWorker runs every five minutes (config/schedule.yml) and calls SendWorkingHoursSummariesService, logging the tally it returns. It honors the instance-wide mobile_push_notifications_dispatch flag, and the built-in drop_sidekiq_jobs_Notifications::MobilePush::WorkingHoursResumeWorker flag can also stop it.
  • The service loads the users whose time has come in one batch of at most 1,000, claims each one from the list right before its push, so a job killed mid-run loses one summary at most and two overlapping runs cannot both send, and tallies the outcome each Gitlab::MobilePush::HeldSummary reports. A deleted user is claimed and dropped so they do not fill the batch forever. A run stops after three minutes under the five-minute cron; what is left waits for the next run.

No new feature flag. Nothing happens until a user turns the setting on, which D enables, and the existing push flags stop pushes and summaries alike.

References

Split from !260440 (closed), as agreed with the reviewer there: seven merge requests of 200 to 300 lines each, cut where the code does not reference the other parts. Each one targets its parent branch and is retargeted to master when the parent merges.

Merge request Targets
A Extract the mobile push recipient checks and the APNs fan-out master
B1 Add a value object for mobile push working hours A
B2 Store mobile push working hours on the user preference B1
C1 Hold mobile pushes for to-dos outside working hours B2
C2a Add the held summary that resumes one user's mobile pushes C1
C2b (this MR) Send the mobile push summaries when working hours begin C2a
D Add the mobile push working hours to the user preferences API C2b

Merge order is the chain order: A, B1, B2, C1, C2a, C2b, then D. D is the switch: it targets C2b, so the API cannot reach master before the server can hold a push and send its summary. Any other order would open a window on GitLab.com where users who enabled the schedule have their pushes held but not summarized until C2b deploys.

Screenshots or screen recordings

No UI changes.

How to set up and validate locally

  1. With a registered device (see doc/api/mobile_push_subscriptions.md) and the mobile_push_notifications_dispatch and mobile_push_notifications feature flags on for your user, set a window that opens a few minutes from now, in a Rails console:
user = User.find_by!(username: 'root')
user.user_preference.update!(
  mobile_push_working_hours_enabled: true,
  mobile_push_working_hours_start: 10 * 60,   # 10:00
  mobile_push_working_hours_end: 17 * 60,     # 17:00
  mobile_push_working_days: %w[monday tuesday wednesday thursday friday],
  mobile_push_working_hours_time_zone: 'Europe/Berlin'
)

  1. From a second user, assign yourself an issue while the window is still closed. No push arrives.

  2. Wait for the window to open (the cron runs every five minutes), or run the worker from the console:

    Notifications::MobilePush::WorkingHoursResumeWorker.new.perform

  3. One push arrives, "You have new to-dos", with the pending badge; log/sidekiq.log shows the worker with delivered: 1.

Database

Per run, three batch reads for the due users, each one index lookup per id, then one pending to-do count per user who passes the recipient checks (item 4). Measured with EXPLAIN ANALYZE on a production clone (postgres.ai, gitlab-production-main, 2026-11-02) with 1,000 synthetic ids (every 12,345th id), the service's batch size and so the worst case for one run. The ids were picked for their count, not their data, so the push-subscriptions read returns no rows; that table holds few rows while the feature rolls out. Rails lists the columns of users and user_preferences explicitly because both models ignore columns; the lists are elided below.

  1. Per summary run, the users: SELECT "users"."id", "users"."email", ... FROM "users" WHERE "users"."id" IN (<1,000 ids>) over users_pkey. 930 rows, 2.607 s cold (I/O read 2.582 s), shared buffers 1,875 hit / 2,034 read. postgres.ai command 560637.

    Plan 
 Index Scan using users_pkey on public.users  (cost=0.44..3161.59 rows=1000 width=1663) (actual time=5.231..2601.159 rows=930 loops=1)
   Index Cond: (users.id = ANY ('{<1,000 ids>}'::integer[]))
   Buffers: shared hit=1875 read=2034 dirtied=15
   WAL: records=19 fpi=15 bytes=113739
   I/O Timings: read=2582.020 write=0.000
Settings: work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB'
Query ID: -8216227850493083091

Time: 2.607 s
  - planning: 5.247 ms
  - execution: 2.602 s
    - I/O read: 2.582 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 1875 (~14.60 MiB) from the buffer pool
  - reads: 2034 (~15.90 MiB) from the OS file cache, including disk I/O
  - dirtied: 15 (~120.00 KiB)
  - writes: 0

  1. Per summary run, their preferences: SELECT "user_preferences"."id", "user_preferences"."user_id", ... FROM "user_preferences" WHERE "user_preferences"."user_id" IN (<1,000 ids>) over index_user_preferences_on_user_id. 747 rows, 1.837 s cold (I/O read 1.817 s), shared buffers 1,894 hit / 1,829 read. postgres.ai command 560639.

    Plan 
 Index Scan using index_user_preferences_on_user_id on public.user_preferences  (cost=0.44..2653.00 rows=1000 width=254) (actual time=8.240..1834.988 rows=747 loops=1)
   Index Cond: (user_preferences.user_id = ANY ('{<1,000 ids>}'::integer[]))
   Buffers: shared hit=1894 read=1829 dirtied=5
   WAL: records=7 fpi=5 bytes=40857
   I/O Timings: read=1817.106 write=0.000
Settings: work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB'
Query ID: 335891706224490731

Time: 1.837 s
  - planning: 1.581 ms
  - execution: 1.836 s
    - I/O read: 1.817 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 1894 (~14.80 MiB) from the buffer pool
  - reads: 1829 (~14.30 MiB) from the OS file cache, including disk I/O
  - dirtied: 5 (~40.00 KiB)
  - writes: 0

  1. Per summary run, their phones: SELECT "mobile_device_push_subscriptions".* FROM "mobile_device_push_subscriptions" WHERE "mobile_device_push_subscriptions"."user_id" IN (<1,000 ids>) over index_mobile_device_push_subscriptions_on_user_id. 0 rows, 4.222 ms cold, shared buffers 0 hit / 1 read. postgres.ai command 560641.

    Plan 
 Index Scan using index_mobile_device_push_subscriptions_on_user_id on public.mobile_device_push_subscriptions  (cost=0.14..6.78 rows=34 width=290) (actual time=1.147..1.148 rows=0 loops=1)
   Index Cond: (mobile_device_push_subscriptions.user_id = ANY ('{<1,000 ids>}'::bigint[]))
   Buffers: shared read=1
   I/O Timings: read=1.083 write=0.000
Settings: work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB'
Query ID: 4920233568043107247

Time: 4.222 ms
  - planning: 2.997 ms
  - execution: 1.225 ms
    - I/O read: 1.083 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 0 from the buffer pool
  - reads: 1 (~8.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

  1. Per due user, the pending to-do count that HeldSummary#resume reads to skip a user with nothing pending and that SummaryPayload#badge sends: User#todos_pending_count, so SELECT COUNT(*) FROM "todos" LEFT JOIN banned_users ON todos.author_id = banned_users.user_id WHERE "todos"."user_id" = <user id> AND "todos"."state" = 'pending' AND "banned_users"."user_id" IS NULL AND "todos"."state" = 'pending' AND ("todos"."snoozed_until" <= '<now>' OR "todos"."snoozed_until" IS NULL) over index_todos_on_user_id_and_id_pending, with an anti join on banned_users_pkey per row. Measured for the user with the most pending to-dos among the last two million rows: 3,513 pending to-dos, 1.180 s cold (I/O read 1.115 s), shared buffers 12,323 hit / 1,707 read; 10.951 ms warm, 14,030 hit / 0 read. postgres.ai commands 561000 (cold) and 561008 (warm).

    Plan (cold) 
 Aggregate  (cost=406.53..406.54 rows=1 width=8) (actual time=1133.348..1133.350 rows=1 loops=1)
   Buffers: shared hit=12323 read=1707 dirtied=51
   WAL: records=55 fpi=51 bytes=411627
   I/O Timings: read=1114.607 write=0.000
   ->  Nested Loop Anti Join  (cost=0.99..406.53 rows=1 width=0) (actual time=3.484..1132.871 rows=3513 loops=1)
         Buffers: shared hit=12323 read=1707 dirtied=51
         WAL: records=55 fpi=51 bytes=411627
         I/O Timings: read=1114.607 write=0.000
         ->  Index Scan using index_todos_on_user_id_and_id_pending on public.todos  (cost=0.56..166.52 rows=122 width=4) (actual time=0.999..1121.936 rows=3513 loops=1)
               Index Cond: (todos.user_id = <user id>)
               Filter: ((todos.snoozed_until <= '<now>'::timestamp with time zone) OR (todos.snoozed_until IS NULL))
               Buffers: shared hit=1785 read=1703 dirtied=51
               WAL: records=55 fpi=51 bytes=411627
               I/O Timings: read=1110.651 write=0.000
         ->  Index Only Scan using banned_users_pkey on public.banned_users  (cost=0.43..1.96 rows=1 width=8) (actual time=0.003..0.003 rows=0 loops=3513)
               Index Cond: (banned_users.user_id = todos.author_id)
               Heap Fetches: 0
               Buffers: shared hit=10538 read=4
               I/O Timings: read=3.956 write=0.000
Settings: work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB'
Query ID: 3541532485087778299

Time: 1.180 s
  - planning: 46.170 ms
  - execution: 1.134 s
    - I/O read: 1.115 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 12323 (~96.30 MiB) from the buffer pool
  - reads: 1707 (~13.30 MiB) from the OS file cache, including disk I/O
  - dirtied: 51 (~408.00 KiB)
  - writes: 0

The three batch reads run once per five-minute run, from a low-urgency cron worker that defer_on_database_health_signal holds back while autovacuum runs on mobile_device_push_subscriptions, the one table it writes, so the 2.6 s cold read of 1,000 user rows is the ceiling for a full batch, not a per-request cost. The count runs once per user who gets as far as the pending check, at most 1,000 per run; User#todos_pending_count caches it for 24 hours, so a user resumed on Monday after a weekend hold is a cache miss. A full batch of heavy users would be about 1,000 index scans like the one above, spread by the service's three-minute run limit over consecutive runs.

The new User.preload_user_preference_and_push_subscriptions scope only preloads; it adds no condition. On a dead device token the existing per-row delete runs, now from Gitlab::MobilePush::Sender:

  1. On a dead device token, the existing per-row DELETE FROM "mobile_device_push_subscriptions" WHERE "mobile_device_push_subscriptions"."id" = $1, now in Gitlab::MobilePush::Sender: 1.361 ms, shared buffers 3 hit / 1 read.

    Plan 
 Delete on public.mobile_device_push_subscriptions  (cost=0.14..3.16 rows=0 width=0) (actual time=0.754..0.755 rows=0 loops=1)
   Buffers: shared hit=3 read=1
   I/O Timings: read=0.710 write=0.000
   ->  Index Scan using mobile_device_push_subscriptions_pkey on public.mobile_device_push_subscriptions  (cost=0.14..3.16 rows=1 width=6) (actual time=0.752..0.753 rows=0 loops=1)
         Index Cond: (mobile_device_push_subscriptions.id = 1)
         Buffers: shared hit=3 read=1
         I/O Timings: read=0.710 write=0.000
Settings: effective_cache_size = '472585MB', work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4'
Query ID: 7132805274627337975

Time: 1.361 ms
  - planning: 0.517 ms
  - execution: 0.844 ms
    - I/O read: 0.710 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 3 (~24.00 KiB) from the buffer pool
  - reads: 1 (~8.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

MR acceptance checklist

Evaluated against the MR acceptance checklist.

🤖 Generated with Claude Code

Edited by Marcel van Remmerden

Merge request reports