Deduplicate authorized_projects queue
This queue can have many duplicate jobs. In #215 (comment 318559897), I outlined how in a recent incident, the vast majority of the jobs were duplicates ('Other' is other jobs with duplicates):
It looks like deduplication would have helped a lot: https://log.gprd.gitlab.net/goto/8e263e9bd209b264f1b245f0ca30bba5
json.duplicate-of.keyword: Descending Count Missing 8,325 3ee914e27ba762352b6a5e75 215 f9630c029fb2c5b8880fd4cb 215 4561ec0d8d9869b474eacc52 26 69cb9d685682384501a1d8ef 25 a1c5f034960be79257230048 25 181fc538e707c2ca6016a612 24 22d7a784d3e3cc049bb10462 24 347b3a92feefbe04b3bb2d47 24 43dde2f6b1859944720d07cc 24 Other 402,834
We enabled deduplication for this queue in #42 (comment 318078109).
We can watch the logs at this (long) Kibana link.
From &176 (comment 318144098):
I think around 08:00 UTC today
We would have seen a higher peak without deduplication:
Edited by Sean McGivern