Runner minute purchase causing issues with quotas
Summary
Today we've received several reports of runner minute quotas behaving badly after purchase of additional minutes. In all cases, the user reports that there is suddenly an unexpected number of minutes used and the additional pool (denominator) is a random number instead of what they purchased.
Additional information from the user w/ 147 denominator:
Purchased 2k minutes today and did not have any job exceed a timeout or anything else notable which may have caused this behavior.
Additional information from the user w/ 4299 denominator:
I'm not sure why using a lot of minutes would cause and issue when we have 4299 remaining minutes when the CI was blocked. I believe there is an issue with how the additional runner minutes calculation is working. We closely monitor the usage daily using scripts so we know how much we have left.
every month we get 2000 minutes. These minutes follow the pattern.
time used/time alotted
and then a percentage is calculated so when we use 100% of our minutes, we're out of our bronze subscription minutes.for additional runner minutes, the format is different.
minutes used/minutes remaining
so you can end up having a situation where you've used 16000 but have 4299 remaining. The math is not the same! you don't want to say "I used more minutes than I have minutes remaining" so I'm going to shut the CI down. We still have minutes that we've not used - we still have remaining minutes. Therefore, shut down the CI when minutes remaining is 0. Calculating a percentage from the additional runner minutes is meaningless.We've been watching this minutes remaining go down over the month and can show you a report of it if you're interested. I checked the report at 7ish AM and there was >4500 minutes left. My best guess is, this is a software issue.
Additional information from the 3rd user (no screenshot available):
We usually get a quota of 10,000 minutes. We saw a warning saying we are within 3% of our monthly quota so I bought 1000 more minutes. Right after the purchase the quota went from 3% left to 235% over. I think it wiped out our 10,000 minutes for the month.
Support Tickets
- https://gitlab.zendesk.com/agent/tickets/130520
- https://gitlab.zendesk.com/agent/tickets/130543
- https://gitlab.zendesk.com/agent/tickets/130529
- https://gitlab.zendesk.com/agent/tickets/130540
- https://gitlab.zendesk.com/agent/tickets/130558
- https://gitlab.zendesk.com/agent/tickets/130561
- https://gitlab.zendesk.com/agent/tickets/130558
- https://gitlab.zendesk.com/agent/tickets/130559
- https://gitlab.zendesk.com/agent/tickets/130581
- https://gitlab.zendesk.com/agent/tickets/130601
- https://gitlab.zendesk.com/agent/tickets/130657
Version Notes
- Aug 19th deployer finished a deployer pipeline of 12.2.201908191220-0c1c17abba9.302fa12ad4f on gprd which took 4 hours and 6 hours (wall time)
- Aug 22nd deployer finished a deployer pipeline of 12.2.201908220421-5b6bdcefa25.2f293fd6cf9 on gprd-cny which took an hour
Investigation notes
Seems unrelated to job timeout gitlab-runner#4147 (closed)
/cc @tipyn @PhilEncarnacion @support-dotcom