[production] Enable `user_refresh_from_replica_worker_uses_replica_db` feature flag

Production Change

Change Summary

user_refresh_from_replica_worker_uses_replica_db is the feature flag that controls whether safety-net project authorization refresh jobs should read from the primary or the replica. This flag is currently OFF in production, which means it reads from the primary.

We need to turn this flag ON, so that these jobs can start reading from the replica.

  • MR introducing this flag: Here
  • Where is this flag used?: Here
  • Rollout Issue: Here
  • Tested on staging?: Yes, staging Kibana dashboard is here and the results are as expected (ie, db_primary_count is 0)

Change Details

  1. Services Impacted - ServiceSidekiq, ServicePatroni, ServicePostgres
  2. Change Technician - @manojmj
  3. Change Reviewer - DRI for the review of this change
  4. Time tracking - 75 minutes
  5. Downtime Component - No downtime expected

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 0 minutes

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 75 minutes

  • /chatops run feature set user_refresh_from_replica_worker_uses_replica_db 10
  • Ensure that this worker is now reading from the replica based on the Kibana query here. Make sure the that db_primary_count is 0 & db_replica_count is not 0 in these documents.
  • /chatops run feature set user_refresh_from_replica_worker_uses_replica_db 25
  • /chatops run feature set user_refresh_from_replica_worker_uses_replica_db 50
  • /chatops run feature set user_refresh_from_replica_worker_uses_replica_db 75
  • /chatops run feature set user_refresh_from_replica_worker_uses_replica_db 100

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - - 1 minute

  • In Kibana dashboard we should see the calls to primary db reducing and calls to replica db increasing

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 0 minutes

  • /chatops run feature set user_refresh_from_replica_worker_uses_replica_db false

Monitoring

Key metrics to observe

  • In Kibana dashboard we should see the calls to primary db reducing and calls to replica db increasing.

  • Sidekiq Queue detail dashboard: Link

  • Postgres Async (Sidekiq) replica Connection Pool Utilization per Node: Link

  • Logs:

    • Link
    • There should not be a lot of host_offline events in the logs

  • Queue length dashboard: Link, queue length should not increase

  • Sidekiq connection pool saturation details: Link

  • Job failure rates: Link, too many jobs should not fail.

Summary of infrastructure changes

  • Does this change introduce new compute instances?
  • Does this change re-size any existing compute instances?
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Summary of the above

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents.
Edited by Manoj M J [OOO]