Geo: ReadOnlySqlTransaction for `User.authorized_projects` in secondary node

Summary

When trying to login as admin in a secondary node on a freshly upgraded system, there may be need to backfill data that was not pre-populated by a previous migration.

We are using a pattern to make this changes whenever it's first needed, which doesn't play well with Geo. While we can't force the data change to happen in a secondary node, we should try to bypass that step and not fail in the user's face because there is missing data.

Steps to reproduce

current user should have admin privileges (to be redirect to admin dashboard) and should have authorized_projects_populated set to false.

What is the current bug behavior?

When trying to access the secondary node, because there are data to be backfilled (probably because of a migration), and the primary node was not accessed before, the secondary node is crashing with a ReadOnlySqlTransaction error.

What is the expected correct behavior?

Update should not happen in a secondary node.

Relevant logs and/or screenshots

Completed 500 Internal Server Error in 55ms (ActiveRecord: 11.5ms | Elasticsearch: 0.0ms)

ActiveRecord::StatementInvalid (PG::ReadOnlySqlTransaction: ERROR:  cannot execute UPDATE in a read-only transaction
: UPDATE "users" SET "authorized_projects_populated" = 't' WHERE "users"."id" = $1):
  app/models/user.rb:549:in `set_authorized_projects_column'
  app/services/users/refresh_authorized_projects_service.rb:81:in `block in update_authorizations'
  app/services/users/refresh_authorized_projects_service.rb:78:in `update_authorizations'
  app/services/users/refresh_authorized_projects_service.rb:68:in `execute_without_lease'
  app/services/users/refresh_authorized_projects_service.rb:41:in `execute'
  app/models/user.rb:540:in `refresh_authorized_projects'
  app/models/user.rb:554:in `authorized_projects'
  app/finders/projects_finder.rb:51:in `init_collection'
  app/finders/projects_finder.rb:30:in `execute'
  app/controllers/dashboard/projects_controller.rb:50:in `load_projects'
  app/controllers/dashboard/projects_controller.rb:8:in `index'
  app/controllers/root_controller.rb:16:in `index'
  app/controllers/application_controller.rb:285:in `set_locale'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:10:in `call'
  lib/gitlab/middleware/readonly_geo.rb:30:in `call'
  lib/gitlab/request_context.rb:18:in `call'

Possible fixes

  def authorized_projects(min_access_level = nil)
    refresh_authorized_projects unless authorized_projects_populated

    # We're overriding an association, so explicitly call super with no arguments or it would be passed as `force_reload` to the association
    projects = super()
    projects = projects.where('project_authorizations.access_level >= ?', min_access_level) if min_access_level

    projects
  end

we should guard the refresh_authorized_projects with an additional unless check:

    refresh_authorized_projects unless authorized_projects_populated || Gitlab::Geo.secondary?

@rspeicher @stanhu @dbalexandre