Multiple notification emails sent for issue comments after upgrade to 13.2

Summary

On Gitlab Omnibus CE. Upgrade from 13.1 to 13.2. After upgrade any comment on an issue (new or existing) sends multiple emails to each user

Steps to reproduce

Upgrade to Gilab CE omnibus 13.2 Create an issue Notification sent based on users settings add a comment Issue Notifications sent to users multiple times (4 times)

Example Project

Unable to reproduce on gitlab.com, this only happens on omnibus CE

What is the current bug behavior?

Issue Notifications sent to users multiple times (4 times)

What is the expected correct behavior?

Issue Notifications sent to users once

Relevant logs and/or screenshots

[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 7bcef9cf-9e5f-476d-a0c0-a7980b7efe9a) to Sidekiq(mailers) with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[2, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [a0583ce7-2bd9-4754-8056-9e0431868ba4] Performing ActionMailer::MailDeliveryJob (Job ID: a0583ce7-2bd9-4754-8056-9e0431868ba4) from Sidekiq(mailers) enqueued at 2020-07-23T15:10:53Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[18, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [43377287-6341-422d-8150-16113608bf96] Performing ActionMailer::MailDeliveryJob (Job ID: 43377287-6341-422d-8150-16113608bf96) from Sidekiq(mailers) enqueued at 2020-07-23T15:10:53Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[3, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [7bcef9cf-9e5f-476d-a0c0-a7980b7efe9a] Delivered mail note_11810@develop.tadpole.cc (1084.2ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [7bcef9cf-9e5f-476d-a0c0-a7980b7efe9a] Performed ActionMailer::MailDeliveryJob (Job ID: 7bcef9cf-9e5f-476d-a0c0-a7980b7efe9a) from Sidekiq(mailers) in 1208.88ms
[ActiveJob] [ActionMailer::MailDeliveryJob] [a0583ce7-2bd9-4754-8056-9e0431868ba4] Delivered mail note_11810@develop.tadpole.cc (1075.4ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [a0583ce7-2bd9-4754-8056-9e0431868ba4] Performed ActionMailer::MailDeliveryJob (Job ID: a0583ce7-2bd9-4754-8056-9e0431868ba4) from Sidekiq(mailers) in 1232.05ms
[ActiveJob] [ActionMailer::MailDeliveryJob] [43377287-6341-422d-8150-16113608bf96] Delivered mail note_11810@develop.tadpole.cc (1059.5ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [43377287-6341-422d-8150-16113608bf96] Performed ActionMailer::MailDeliveryJob (Job ID: 43377287-6341-422d-8150-16113608bf96) from Sidekiq(mailers) in 1227.91ms
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: e6d4c836-baff-451f-874f-eb410f952880) to Sidekiq(mailers) with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[2, 11810, nil]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 069b6462-2189-428c-a6cb-5c36eac75eda) to Sidekiq(mailers) with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[18, 11810, nil]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 5142202c-2509-4b20-a192-db9f38bfaa3e) to Sidekiq(mailers) with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[3, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [069b6462-2189-428c-a6cb-5c36eac75eda] Performing ActionMailer::MailDeliveryJob (Job ID: 069b6462-2189-428c-a6cb-5c36eac75eda) from Sidekiq(mailers) enqueued at 2020-07-23T15:11:38Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[18, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [e6d4c836-baff-451f-874f-eb410f952880] Performing ActionMailer::MailDeliveryJob (Job ID: e6d4c836-baff-451f-874f-eb410f952880) from Sidekiq(mailers) enqueued at 2020-07-23T15:11:38Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[2, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [5142202c-2509-4b20-a192-db9f38bfaa3e] Performing ActionMailer::MailDeliveryJob (Job ID: 5142202c-2509-4b20-a192-db9f38bfaa3e) from Sidekiq(mailers) enqueued at 2020-07-23T15:11:38Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[3, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [e6d4c836-baff-451f-874f-eb410f952880] Delivered mail note_11810@develop.tadpole.cc (1045.4ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [e6d4c836-baff-451f-874f-eb410f952880] Performed ActionMailer::MailDeliveryJob (Job ID: e6d4c836-baff-451f-874f-eb410f952880) from Sidekiq(mailers) in 1213.2ms
[ActiveJob] [ActionMailer::MailDeliveryJob] [069b6462-2189-428c-a6cb-5c36eac75eda] Delivered mail note_11810@develop.tadpole.cc (1065.2ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [069b6462-2189-428c-a6cb-5c36eac75eda] Performed ActionMailer::MailDeliveryJob (Job ID: 069b6462-2189-428c-a6cb-5c36eac75eda) from Sidekiq(mailers) in 1255.05ms
[ActiveJob] [ActionMailer::MailDeliveryJob] [5142202c-2509-4b20-a192-db9f38bfaa3e] Delivered mail note_11810@develop.tadpole.cc (1047.2ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [5142202c-2509-4b20-a192-db9f38bfaa3e] Performed ActionMailer::MailDeliveryJob (Job ID: 5142202c-2509-4b20-a192-db9f38bfaa3e) from Sidekiq(mailers) in 1258.92ms
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 6835a724-15df-4346-aa57-2e35cfaa9ea7) to Sidekiq(mailers) with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[2, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [6835a724-15df-4346-aa57-2e35cfaa9ea7] Performing ActionMailer::MailDeliveryJob (Job ID: 6835a724-15df-4346-aa57-2e35cfaa9ea7) from Sidekiq(mailers) enqueued at 2020-07-23T15:12:36Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[2, 11810, nil]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: de787eaf-ecea-44cc-ae53-46ccbdd6d082) to Sidekiq(mailers) with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[18, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [6ca2bd6d-d369-492b-9c50-1379d8b1838b] Performing ActionMailer::MailDeliveryJob (Job ID: 6ca2bd6d-d369-492b-9c50-1379d8b1838b) from Sidekiq(mailers) enqueued at 2020-07-23T15:12:36Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[3, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [de787eaf-ecea-44cc-ae53-46ccbdd6d082] Performing ActionMailer::MailDeliveryJob (Job ID: de787eaf-ecea-44cc-ae53-46ccbdd6d082) from Sidekiq(mailers) enqueued at 2020-07-23T15:12:36Z with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[18, 11810, nil]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 6ca2bd6d-d369-492b-9c50-1379d8b1838b) to Sidekiq(mailers) with arguments: "Notify", "note_issue_email", "deliver_now", {:args=>[3, 11810, nil]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [de787eaf-ecea-44cc-ae53-46ccbdd6d082] Delivered mail note_11810@develop.tadpole.cc (1049.3ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [de787eaf-ecea-44cc-ae53-46ccbdd6d082] Performed ActionMailer::MailDeliveryJob (Job ID: de787eaf-ecea-44cc-ae53-46ccbdd6d082) from Sidekiq(mailers) in 1159.49ms
[ActiveJob] [ActionMailer::MailDeliveryJob] [6835a724-15df-4346-aa57-2e35cfaa9ea7] Delivered mail note_11810@develop.tadpole.cc (1087.2ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [6835a724-15df-4346-aa57-2e35cfaa9ea7] Performed ActionMailer::MailDeliveryJob (Job ID: 6835a724-15df-4346-aa57-2e35cfaa9ea7) from Sidekiq(mailers) in 1234.4ms
[ActiveJob] [ActionMailer::MailDeliveryJob] [6ca2bd6d-d369-492b-9c50-1379d8b1838b] Delivered mail note_11810@develop.tadpole.cc (1067.6ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [6ca2bd6d-d369-492b-9c50-1379d8b1838b] Performed ActionMailer::MailDeliveryJob (Job ID: 6ca2bd6d-d369-492b-9c50-1379d8b1838b) from Sidekiq(mailers) in 1227.11ms

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Debian 10
Current User:   git
Using RVM:      no
Ruby Version:   2.6.6p146
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.9
Git Version:    2.27.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.2.0
Revision:       d47fc5085a7
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.7
URL:            https://develop.tadpole.cc
HTTP Clone URL: https://develop.tadpole.cc/some-group/some-project.git
SSH Clone URL:  git@develop.tadpole.cc:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: github

GitLab Shell
Version:        13.3.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking ...
yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
9/2 ... yes
9/3 ... yes
7/4 ... yes
16/7 ... yes
8/8 ... yes
7/9 ... yes
8/10 ... yes
8/11 ... yes
7/12 ... yes
8/13 ... yes
8/14 ... yes
7/15 ... yes
16/16 ... yes
8/17 ... yes
7/18 ... yes
8/19 ... yes
16/20 ... yes
9/21 ... yes
9/22 ... yes
9/24 ... yes
9/25 ... yes
16/26 ... yes
16/27 ... yes
16/28 ... yes
16/29 ... yes
7/31 ... yes
7/32 ... yes
7/33 ... yes
8/34 ... yes
16/35 ... yes
7/36 ... yes
16/37 ... yes
8/39 ... yes
8/41 ... yes
7/42 ... yes
16/43 ... yes
8/44 ... yes
8/47 ... yes
7/48 ... yes
16/49 ... yes
8/50 ... yes
8/51 ... yes
8/52 ... yes
8/53 ... yes
8/54 ... yes
8/55 ... yes
7/56 ... yes
8/57 ... yes
8/58 ... yes
8/59 ... yes
7/60 ... yes
8/61 ... yes
16/62 ... yes
16/63 ... yes
8/64 ... yes
16/65 ... yes
8/66 ... yes
7/67 ... yes
8/68 ... yes
8/69 ... yes
8/70 ... yes
8/71 ... yes
8/72 ... yes
8/73 ... yes
8/74 ... yes
8/75 ... yes
8/76 ... yes
16/77 ... yes
7/78 ... yes
8/79 ... yes
8/80 ... yes
7/81 ... yes
8/82 ... yes
7/83 ... yes
8/84 ... yes
8/85 ... yes
8/87 ... yes
8/88 ... yes
8/89 ... yes
8/90 ... yes
7/91 ... yes
8/92 ... yes
8/93 ... yes
8/94 ... yes
8/95 ... yes
8/101 ... yes
7/102 ... yes
7/109 ... yes
7/115 ... yes
9/116 ... yes
9/117 ... yes
7/118 ... yes
8/119 ... yes
8/120 ... yes
7/121 ... yes
7/122 ... yes
17/123 ... yes
8/125 ... yes
10/126 ... yes
7/127 ... yes
7/128 ... yes
17/129 ... yes
17/130 ... yes
17/131 ... yes
7/132 ... yes
8/133 ... yes
8/134 ... yes
16/135 ... yes
8/136 ... yes
8/138 ... yes
7/139 ... yes
19/140 ... yes
7/141 ... yes
16/142 ... yes
8/143 ... yes
9/144 ... yes
7/145 ... yes
17/146 ... yes
7/147 ... yes
7/148 ... yes
8/149 ... yes
7/152 ... yes
8/153 ... yes
7/154 ... yes
7/155 ... yes
16/156 ... yes
9/158 ... yes
9/159 ... yes
8/160 ... yes
8/161 ... yes
8/162 ... yes
8/163 ... yes
8/164 ... yes
7/165 ... yes
9/166 ... yes
7/167 ... yes
2/168 ... yes
8/169 ... yes
25/170 ... yes
8/171 ... yes
17/172 ... yes
8/173 ... yes
7/174 ... yes
8/175 ... yes
7/176 ... yes
7/177 ... yes
8/178 ... yes
8/179 ... yes
7/180 ... yes
16/181 ... yes
7/182 ... yes
16/183 ... yes
9/184 ... yes
29/185 ... yes
29/186 ... yes
29/187 ... yes
29/188 ... yes
29/189 ... yes
29/190 ... yes
29/191 ... yes
29/192 ... yes
30/193 ... yes
30/194 ... yes
30/195 ... yes
30/196 ... yes
30/197 ... yes
30/198 ... yes
30/199 ... yes
32/200 ... yes
32/201 ... yes
9/202 ... yes
7/203 ... yes
10/204 ... yes
17/205 ... yes
33/206 ... yes
8/207 ... yes
7/208 ... yes
7/209 ... yes
9/210 ... yes
7/211 ... yes
8/212 ... yes
9/213 ... yes
8/214 ... yes
8/215 ... yes
17/216 ... yes
9/217 ... yes
7/220 ... yes
3/221 ... yes
2/223 ... yes
7/224 ... yes
7/225 ... yes
7/226 ... yes
8/227 ... yes
7/228 ... yes
7/229 ... yes
7/230 ... yes
17/231 ... yes
17/232 ... yes
17/233 ... yes
7/234 ... yes
17/235 ... yes
7/236 ... yes
16/237 ... yes
16/240 ... yes
7/241 ... yes
7/242 ... yes
7/244 ... yes
7/245 ... yes
7/246 ... yes
29/249 ... yes
7/250 ... yes
29/251 ... yes
9/252 ... yes
9/253 ... yes
10/254 ... yes
7/255 ... yes
7/256 ... yes
17/257 ... yes
8/258 ... yes
8/259 ... yes
5/260 ... yes
7/261 ... yes
81/262 ... yes
113/263 ... yes
7/264 ... yes
107/265 ... yes
86/266 ... yes
114/267 ... yes
10/268 ... yes
8/269 ... yes
7/270 ... yes
7/271 ... yes
8/273 ... yes
7/274 ... yes
10/275 ... yes
10/276 ... yes
3/277 ... yes
101/278 ... yes
10/279 ... yes
7/280 ... yes
10/282 ... yes
10/284 ... yes
7/285 ... yes
2/286 ... yes
9/287 ... yes
7/288 ... yes
9/289 ... yes
8/290 ... yes
8/291 ... yes
7/292 ... yes
10/293 ... yes
10/294 ... yes
7/295 ... yes
8/296 ... yes
7/297 ... yes
8/298 ... yes
9/299 ... yes
9/300 ... yes
10/301 ... yes
85/302 ... yes
9/303 ... yes
9/304 ... yes
9/305 ... yes
17/306 ... yes
17/307 ... yes
7/308 ... yes
8/309 ... yes
8/310 ... yes
105/311 ... yes
8/312 ... yes
7/313 ... yes
7/314 ... yes
10/315 ... yes
7/316 ... yes
10/317 ... yes
10/318 ... yes
16/319 ... yes
42/320 ... yes
9/321 ... yes
9/322 ... yes
9/323 ... yes
7/324 ... yes
7/325 ... yes
7/326 ... yes
7/327 ... yes
7/328 ... yes
42/329 ... yes
42/330 ... yes
81/335 ... yes
81/336 ... yes
81/337 ... yes
9/339 ... yes
83/340 ... yes
83/341 ... yes
32/342 ... yes
7/343 ... yes
29/344 ... yes
29/345 ... yes
85/346 ... yes
86/347 ... yes
86/348 ... yes
17/349 ... yes
7/350 ... yes
88/351 ... yes
88/352 ... yes
88/353 ... yes
88/354 ... yes
88/355 ... yes
88/356 ... yes
88/357 ... yes
7/359 ... yes
90/360 ... yes
93/361 ... yes
97/362 ... yes
93/363 ... yes
98/364 ... yes
2/365 ... yes
8/366 ... yes
7/367 ... yes
93/368 ... yes
93/369 ... yes
101/370 ... yes
7/371 ... yes
7/372 ... yes
103/373 ... yes
103/374 ... yes
107/375 ... yes
109/376 ... yes
113/377 ... yes
109/378 ... yes
114/379 ... yes
114/380 ... yes
7/381 ... yes
115/382 ... yes
115/383 ... yes
7/384 ... yes
7/385 ... yes
7/386 ... yes
25/387 ... yes
8/388 ... yes
8/389 ... yes
7/390 ... yes
7/391 ... yes
2/393 ... yes
7/394 ... yes
2/397 ... yes
2/398 ... yes
115/399 ... yes
8/400 ... yes
93/401 ... yes
113/402 ... yes
114/403 ... yes
10/404 ... yes
83/405 ... yes
126/406 ... yes
115/407 ... yes
113/408 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.27.0)
Git user has default SSH configuration? ... yes
Active users: ... 88
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)