Skip to content

gitlab-rails: check dependencies directly instead of via Rake call

Jason Plum requested to merge charts-1775-alternate-wait-for-deps into master

Summary

Address gitlab-org/charts/gitlab#1775 (closed)'s observation of the amount of time needed to complete init, by replacing calls to Rake with direct calls to the resources we need to check (Redis / PostgreSQL)

Indicators of "complete"

  • Redis
    • Supports Sentinel
    • Supports rediss://
    • Checks all configured Redis servers
    • Ruby only, as this covers "all" bases of connectivity support within Rails
  • PostgreSQL
  • (Re)starts Ruby as few times as possible, loading as few modules as possible.

Details

Implements the basic concept derived from this comment, which is to directly query the various systems.

  1. Add scripts to directly access PostgreSQL and Redis dependencies
    • rails-db-check: Ruby script with absolute minimum viability, limited to only checking the connection, and the state of the migrations according to ActiveRecord.
    • rails-redis-ping: uses the Redis gem to directly connect and "ping" redis, input with resque.yml and variants.
  2. Update wait-for-deps script to call direct access scripts instead of calling rake.

Impact

Without the need to spin up Rails via the Rake call, we can make a quite large impact on the time it takes to find out if we're "ready".

How big? Observe the below, performed against operational deployments (e.g. "shortest possible"), that have had Rake commands run several times:

git@gke-production-a4b9oa-task-runner-77fcf8cb79-8lgpf:/$ time /scripts/wait-for-deps 
Checking database connection and schema version
Database Schema - current: 20200407094923, codebase: 20200407094923

real    0m11.174s
user    0m9.497s
sys     0m1.335s
git@gitlab-task-runner-7459858f85-9cwf7:/srv/gitlab$  time /scripts/wait-for-deps
Checking database connectivity and schema version
Database Schema - current: 20200511145545, codebase: 20200511145545
Checking Redis connectivity
Checking 'resque.yml'
- Attempting to connect to 'redis://gitlab-redis-master:6379' SUCCESS

real    0m3.459s
user    0m2.010s
sys     0m1.367s
Former, with use of `psql -c` directly
git@demo-task-runner-74b4679696-vvpxl:/srv/gitlab$ time /scripts/wait-for-deps
Checking database connection and schema version
Database Schema - current: 20200313123934, codebase: 20200313123934
Checking Redis connectivity
  - checking resque.yml
    GOOD

real 0m1.366s user 0m1.034s sys 0m0.309s

cc @skarbek

Closes gitlab-org/charts/gitlab#1775 (closed)

Edited by Robert Speicher

Merge request reports