Ci::RunnerManager.version is not being written out from the Redis cache

As seen in the tests for a previous issue, Ci::RunnerManager#heartbeat is somehow not writing out the version fields that are being set from the client:

We can see that the handcrafted runner versions are correctly computed and inserted into the ci_runner_versions table:

However, the ci_runner_machines.version column is not being updated in the respective records even after 4 hours. It seems to never have been written from Redis into the table after the usual 40-55 minutes:

This seems to have happened because the runner managers went offline and didn't continue to ping for jobs. The records were unloaded from memory without the cached values being written to the database. When I restarted one of the runner managers (3403321), it immediately wrote the values that had been cached in Redis.

Proposal

Create a cron worker that periodically syncs the Redis data to the database, after the data has been in memory for 1 hour. This is outside the 40-55 minutes range that is allowed for online runners to write their data to the table.

Implementation plan

1. Create a cron job that runs hourly, and checks in the Redis cache for cached Ci::Runner attributes:

module Ci
  class Runner
    def self.runner_cached_ids
      Gitlab::Redis::SharedState.with { |r| r.keys("cache:#{self.name}:*:attributes").map { |cache_id| cache_id.split(':')[-2].to_i } }
    end
  end
end

1. Load the respective models:

runners = Ci::Runner.id_in(Ci::Runner.runner_cached_ids)

1. For each element in runners, call update_columns(values) if persist_cached_data?