Disable partition checking and creation on application startup
C4
Production Change - Criticality 4Change Component | Description |
---|---|
Change Objective | Disable partition checking and creation on application startup |
Change Type | ConfigurationChange |
Services Impacted | Rails |
Change Team Members | tbd |
Change Criticality | C4 |
Change Reviewer or tested in staging | Staging |
Due Date | - |
Time tracking | - |
Background
We introduced dynamic postgres partitions into the application and started using it. Those are "dynamic" because they are being created at runtime. Therefore, we have a couple places where we need to check if new partitions need to be created: There is a sidekiq cron (6 hourly) and we also must have this in the application initializer.
However, for the GitLab.com environment we have two reasons not to do this in the initializer:
- The application runs 24/7 anyways - so the cron is enough to do the work
- On restarting the app (e.g. during deploy), the initializer executes. There is absolutely no need the whole fleet executes this checking.
Therefore, we added an environment variable we can configure on GitLab.com such that this initializer is being skipped. We only rely on the sidekiq cron. Partitions are being created well in advance of actually needing them.
For more details, please see gitlab-org/gitlab!35981 (merged). The env flag has been added with gitlab-org/gitlab!37358 (diffs).
Detailed steps for the change
Configure the Rails environment to have the enironment variable DISABLE_POSTGRES_PARTITION_CREATION_ON_STARTUP=1
. This applies to all instances starting and running the Rails application (e.g. web, sidekiq and others).
In order for the change to take effect, a application restart is needed. Unless there are compelling reasons to do so, we don't need to force restart the application for this change and from an app perspective, it's ok to wait until the next deploy happens.
Rollback steps
Undo the configuration change, so that DISABLE_POSTGRES_PARTITION_CREATION_ON_STARTUP
env variable is entirely absent from the environment.
Again, a force restart should not be necessary.
Monitoring
Key metrics to observe
There are no metrics to observe.
We log when the partition checking is being triggered: https://log.gprd.gitlab.net/goto/f6b0c22c5dcc4e4d808fd6739d8cc849
Before this change, we see invocations from the sideiq cron and bursts of invocations from when the fleet restarts (on deploy).
After this change, we should only see one invocation for this per 6 hours (i.e. 4 times a day) - those come from the sidekiq cron.
Verification
Start a rails console on any of the targeted hosts and check the env variable is present:
$ ENV['DISABLE_POSTGRES_PARTITION_CREATION_ON_STARTUP']
=> 1
Summary of infrastruture changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Nothing like that - only an application configuration change.
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
SRE on-call has been informed prior to change being rolled out -
There are currently no active incidents