integration services have multiple recodrs in services table

... and are flooding enormousely.

It was a 11.11.3 here and I've enabled mattermost service template on Friday. it worked like a charm and I've headed home.

On Monday, I've discovered that gitlab instance ate all 8 cpu heads up to 100% and mattermost notification channel is flooded with similar messages.

sidekiq was full of retries, and I've headed to postgres (not sure why)

after set statement_timeout = '1000 s'; I've finally managed to get an answer.

select count(*) from services where "services"."project_id" = 392 and type='MattermostService';
 count  
--------
 232532
(1 row)

and for the whole instance it was over twenty three millions of records (and I have not more than 400 projects now)!

I'm removing them now, but I'm not sure, what the heck happened and when??

any hints are appreciated!

P.S.: I'm using official omnibus docker image and now upgraded to 12.3.5 (2417d5be)