Monitor DigitalOcean token rate limits

We use DigitalOcean tokens to provision new machines. It's possible that we can run out of tokens and then we will start seeing increased errors for provisioning machines.

Currently for checking limits I use the following script:

checkLimits() {
	echo "$1..."
	curl --silent --head -H "Content-Type: application/json" -H "Authorization: Bearer $2" "https://api.digitalocean.com/v2/" | grep RateLimit
}

checkLimits docker-ci-1-gitlab-com TOKEN

Which outputs:

docker-ci-1-gitlab-com...
RateLimit-Limit: 20000
RateLimit-Remaining: 19322
RateLimit-Reset: 1466080338

We are interested in drawing RateLimit-Remaining. Since the curl uses request themselves we should probably ask for RateLimit every 1m to 3m.

The threshold should be set (we currently have 20k tokens) to something like below 5k.

We could monitor this tokens in context of machine that is using them, instead of doing that centrally. The tokens on that machine you can check in /etc/gitlab-runner/config.toml. There's digitalocean-access-token=.

@maratkalibek What do you think?