Retry internal_id lock instead of blocking the query
This is an extract from https://gitlab.com/gitlab-org/gitlab-ce/issues/54270#note_135909461, below is a summary.
Currently, while we increment internal ids, we would SELECT ... FOR UPDATE the record in internal_ids. Until the lock is acquired, the query would block. If this runs into a statement timeout, the resulting error is non-recoverable at the moment (transaction abort).
The suggestion is to use SELECT ... FOR UPDATE SKIP LOCKED instead. If this returns 0 rows, we would have to retry (with backoff) until we successfully obtain the lock. This makes sure that there are no long running queries that solely wait on a lock and eventually fail due to timeout.