Ci::RunnerManager.version is not being written out from the Redis cache
As seen in the tests for a previous issue, Ci::RunnerManager#heartbeat is somehow not writing out the version
fields that are being set from the client:
We can see that the handcrafted runner versions are correctly computed and inserted into the ci_runner_versions
table:
However, the ci_runner_machines.version
column is not being updated in the respective records even after 4 hours. It seems to never have been written from Redis into the table after the usual 40-55 minutes:
This seems to have happened because the runner managers went offline and didn't continue to ping for jobs. The records were unloaded from memory without the cached values being written to the database. When I restarted one of the runner managers (3403321
), it immediately wrote the values that had been cached in Redis.
Proposal
Create a cron worker that periodically syncs the Redis data to the database, after the data has been in memory for 1 hour. This is outside the 40-55 minutes range that is allowed for online runners to write their data to the table.
Implementation plan
- Create a cron job that runs hourly, and checks in the Redis cache for cached
Ci::Runner
attributes:
module Ci
class Runner
def self.runner_cached_ids
Gitlab::Redis::SharedState.with { |r| r.keys("cache:#{self.name}:*:attributes").map { |cache_id| cache_id.split(':')[-2].to_i } }
end
end
end
- Load the respective models:
runners = Ci::Runner.id_in(Ci::Runner.runner_cached_ids)
- For each element in
runners
, callupdate_columns(values) if persist_cached_data?