Request Group or Project membershis doesn't send reliably email

Summary

Many nested groups with internal visibility.

In the leaf groups are one or more projects with private visibility.

User authenticate against ldap.

Users, who want to acces projects, request membership to the respective group.

This should trigger emails to owners and / or maintainers of the group.

This does not happen reliably, actually it just happens from time to time.

Gitlab delivers mail through a locally installed postfix without authentication and tls.

We observe:

• in sidekiq a changing number of delivery jobs are created. This seems to be indeterministically and not connected to the actual number of maintainers or owners.

• even when sidekiq reports a mail delivery job as "done", no connection request to the local mta. (even tcpdump doesn's show anything, and thus a misconfigured postfix wouldn't affect the result at all)

Steps to reproduce

• Create a new group

• Create a new user

• Log in as the user

• navigate to the newly created group

• request membership to that group

(How one can reproduce the issue - this is very important)

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

• mails to the maintainers and / or owners are not sent
• some times a non empty subset of the owners get's the mail; when the postfix mailserver get's the request to deliver the email, it is actually delivered.
• most of the times no attempt to deliver mails happens
• the number of Jobs in sidekiq never tries to deliver to all maintaners / owners (always less than 10)

What is the expected correct behavior?

• Deliver Mails to up to 10 maintainers or owners of the respective group.

(What you should see instead)

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)
sudo gitlab-rake gitlab:env:info

System information
System:		Ubuntu 18.04
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.3.0
Revision:	9119206b6e8
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://gitlab.seeburger.de
HTTP Clone URL:	https://gitlab.seeburger.de/some-group/some-project.git
SSH Clone URL:	git@gitlab.seeburger.de:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.6.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
34/26 ... yes
351/641 ... yes
501/981 ... yes
501/982 ... yes
504/983 ... yes
549/986 ... yes
518/1005 ... yes
519/1006 ... yes
519/1007 ... yes
520/1008 ... yes
521/1009 ... yes
522/1010 ... yes
523/1011 ... yes
524/1012 ... yes
525/1013 ... yes
525/1014 ... yes
525/1015 ... yes
525/1016 ... yes
525/1017 ... yes
525/1018 ... yes
526/1019 ... yes
527/1020 ... yes
527/1021 ... yes
527/1022 ... yes
527/1023 ... yes
527/1024 ... yes
528/1025 ... yes
528/1026 ... yes
529/1027 ... yes
530/1028 ... yes
531/1029 ... yes
532/1030 ... yes
533/1031 ... yes
534/1032 ... yes
535/1033 ... yes
537/1034 ... yes
537/1035 ... yes
537/1036 ... yes
537/1037 ... yes
537/1038 ... yes
537/1039 ... yes
537/1040 ... yes
537/1041 ... yes
537/1042 ... yes
537/1043 ... yes
537/1044 ... yes
537/1045 ... yes
537/1046 ... yes
537/1047 ... yes
537/1048 ... yes
537/1049 ... yes
537/1050 ... yes
537/1051 ... yes
538/1052 ... yes
539/1053 ... yes
540/1054 ... yes
541/1055 ... yes
542/1056 ... yes
543/1057 ... yes
544/1058 ... yes
545/1059 ... yes
546/1060 ... yes
547/1061 ... yes
548/1062 ... yes
549/1063 ... yes
550/1064 ... yes
551/1065 ... yes
552/1066 ... yes
553/1067 ... yes
553/1068 ... yes
553/1069 ... yes
554/1070 ... yes
555/1071 ... yes
556/1072 ... yes
556/1073 ... yes
557/1074 ... yes
557/1075 ... yes
557/1076 ... yes
558/1077 ... yes
559/1078 ... yes
560/1079 ... yes
560/1080 ... yes
560/1081 ... yes
561/1082 ... yes
561/1083 ... yes
562/1084 ... yes
563/1085 ... yes
564/1086 ... yes
564/1087 ... yes
565/1088 ... yes
566/1089 ... yes
567/1090 ... yes
567/1091 ... yes
567/1092 ... yes
568/1093 ... yes
569/1094 ... yes
570/1095 ... yes
570/1096 ... yes
570/1097 ... yes
571/1098 ... yes
572/1099 ... yes
572/1100 ... yes
572/1101 ... yes
572/1102 ... yes
573/1103 ... yes
574/1104 ... yes
575/1105 ... yes
575/1106 ... yes
575/1107 ... yes
575/1108 ... yes
575/1109 ... yes
575/1110 ... yes
576/1111 ... yes
576/1112 ... yes
577/1113 ... yes
577/1114 ... yes
577/1115 ... yes
577/1116 ... yes
577/1117 ... yes
577/1118 ... yes
577/1119 ... yes
577/1120 ... yes
577/1121 ... yes
577/1122 ... yes
578/1123 ... yes
579/1124 ... yes
579/1125 ... yes
579/1126 ... yes
580/1127 ... yes
581/1128 ... yes
581/1129 ... yes
603/1130 ... yes
632/1131 ... yes
583/1132 ... yes
584/1133 ... yes
585/1134 ... yes
586/1135 ... yes
587/1136 ... yes
588/1137 ... yes
589/1138 ... yes
590/1139 ... yes
591/1140 ... yes
592/1141 ... yes
593/1142 ... yes
594/1143 ... yes
594/1144 ... yes
595/1145 ... yes
595/1146 ... yes
595/1147 ... yes
595/1148 ... yes
596/1149 ... yes
596/1150 ... yes
597/1151 ... yes
598/1152 ... yes
598/1153 ... yes
599/1154 ... yes
599/1155 ... yes
599/1156 ... yes
599/1157 ... yes
600/1158 ... yes
601/1159 ... yes
602/1160 ... yes
602/1161 ... yes
602/1162 ... yes
602/1163 ... yes
603/1164 ... yes
603/1165 ... yes
603/1166 ... yes
603/1167 ... yes
603/1168 ... yes
603/1169 ... yes
603/1170 ... yes
604/1171 ... yes
605/1172 ... yes
606/1173 ... yes
607/1174 ... yes
607/1175 ... yes
608/1176 ... yes
609/1177 ... yes
610/1178 ... yes
611/1179 ... yes
612/1180 ... yes
612/1181 ... yes
613/1182 ... yes
614/1183 ... yes
615/1184 ... yes
616/1185 ... yes
617/1186 ... yes
618/1187 ... yes
619/1188 ... yes
620/1189 ... yes
621/1190 ... yes
621/1191 ... yes
621/1192 ... yes
621/1193 ... yes
622/1194 ... yes
622/1195 ... yes
622/1196 ... yes
622/1197 ... yes
622/1198 ... yes
623/1199 ... yes
624/1200 ... yes
624/1201 ... yes
625/1202 ... yes
626/1203 ... yes
626/1204 ... yes
626/1205 ... yes
627/1206 ... yes
628/1207 ... yes
628/1208 ... yes
628/1209 ... yes
628/1210 ... yes
629/1211 ... yes
630/1212 ... yes
631/1213 ... yes
632/1214 ... yes
632/1215 ... yes
632/1216 ... yes
554/1217 ... yes
632/1218 ... yes
632/1219 ... yes
632/1220 ... yes
632/1221 ... yes
663/1222 ... yes
632/1223 ... yes
632/1224 ... yes
632/1225 ... yes
632/1226 ... yes
632/1227 ... yes
632/1228 ... yes
705/1229 ... yes
632/1230 ... yes
632/1231 ... yes
633/1232 ... yes
6/1233 ... yes
635/1234 ... yes
636/1235 ... yes
637/1236 ... yes
638/1237 ... yes
639/1238 ... yes
639/1239 ... yes
640/1240 ... yes
641/1241 ... yes
641/1242 ... yes
641/1243 ... yes
641/1244 ... yes
642/1245 ... yes
643/1246 ... yes
643/1247 ... yes
644/1248 ... yes
645/1249 ... yes
645/1250 ... yes
645/1251 ... yes
646/1252 ... yes
647/1253 ... yes
648/1254 ... yes
649/1255 ... yes
650/1256 ... yes
651/1257 ... yes
652/1258 ... yes
653/1259 ... yes
653/1260 ... yes
653/1261 ... yes
654/1262 ... yes
655/1263 ... yes
656/1264 ... yes
656/1265 ... yes
656/1266 ... yes
657/1267 ... yes
658/1268 ... yes
659/1269 ... yes
660/1270 ... yes
661/1271 ... yes
661/1272 ... yes
661/1273 ... yes
661/1274 ... yes
661/1275 ... yes
661/1276 ... yes
661/1277 ... yes
661/1278 ... yes
661/1279 ... yes
661/1280 ... yes
661/1281 ... yes
661/1282 ... yes
661/1283 ... yes
661/1284 ... yes
661/1285 ... yes
661/1286 ... yes
662/1287 ... yes
662/1288 ... yes
662/1289 ... yes
663/1290 ... yes
663/1291 ... yes
664/1292 ... yes
664/1293 ... yes
665/1294 ... yes
665/1295 ... yes
666/1296 ... yes
667/1297 ... yes
668/1298 ... yes
669/1299 ... yes
669/1300 ... yes
669/1301 ... yes
669/1302 ... yes
669/1303 ... yes
669/1304 ... yes
669/1305 ... yes
669/1306 ... yes
669/1307 ... yes
669/1308 ... yes
669/1309 ... yes
669/1310 ... yes
669/1311 ... yes
669/1312 ... yes
670/1313 ... yes
670/1314 ... yes
670/1315 ... yes
670/1316 ... yes
670/1317 ... yes
670/1318 ... yes
670/1319 ... yes
670/1320 ... yes
670/1321 ... yes
670/1322 ... yes
670/1323 ... yes
670/1324 ... yes
670/1325 ... yes
670/1326 ... yes
670/1327 ... yes
670/1328 ... yes
670/1329 ... yes
670/1330 ... yes
670/1331 ... yes
670/1332 ... yes
670/1333 ... yes
670/1334 ... yes
670/1335 ... yes
670/1336 ... yes
670/1337 ... yes
670/1338 ... yes
670/1339 ... yes
670/1340 ... yes
670/1341 ... yes
670/1342 ... yes
670/1343 ... yes
670/1344 ... yes
670/1345 ... yes
671/1346 ... yes
671/1347 ... yes
672/1348 ... yes
673/1349 ... yes
673/1350 ... yes
673/1351 ... yes
674/1352 ... yes
675/1353 ... yes
676/1354 ... yes
677/1355 ... yes
677/1356 ... yes
678/1357 ... yes
679/1358 ... yes
680/1359 ... yes
681/1360 ... yes
682/1361 ... yes
683/1362 ... yes
683/1363 ... yes
684/1364 ... yes
685/1365 ... yes
685/1366 ... yes
685/1367 ... yes
685/1368 ... yes
685/1369 ... yes
685/1370 ... yes
685/1371 ... yes
685/1372 ... yes
685/1373 ... yes
685/1374 ... yes
685/1375 ... yes
686/1376 ... yes
687/1377 ... yes
687/1378 ... yes
688/1379 ... yes
688/1380 ... yes
689/1381 ... yes
690/1382 ... yes
691/1383 ... yes
692/1384 ... yes
693/1385 ... yes
694/1386 ... yes
695/1387 ... yes
695/1388 ... yes
695/1389 ... yes
695/1390 ... yes
696/1391 ... yes
696/1392 ... yes
696/1393 ... yes
696/1394 ... yes
697/1395 ... yes
697/1396 ... yes
697/1397 ... yes
698/1398 ... yes
699/1399 ... yes
699/1400 ... yes
700/1401 ... yes
621/1402 ... yes
670/1403 ... yes
600/1404 ... yes
705/1405 ... yes
694/1406 ... yes
600/1407 ... yes
706/1408 ... yes
706/1409 ... yes
706/1410 ... yes
537/1411 ... yes
537/1412 ... yes
537/1413 ... yes
701/1414 ... yes
537/1415 ... yes
537/1416 ... yes
537/1417 ... yes
537/1418 ... yes
537/1419 ... yes
537/1420 ... yes
537/1421 ... yes
670/1422 ... yes
520/1423 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 31
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)