`SQLDataStore`: add configuration parameters for automatic job pruning
Before raising this MR, consider whether the following are required, and complete if so:
[ ] Unit tests[ ] Metrics-
Documentation update(s) TODO
If not required, please explain in brief why not.
Description
This extends the schema for sql-scheduler
with options needed for the cleanup mechanism (#367 (closed)). That is:
-
max_job_age
: maximum time that a job should be kept in the database after it was marked as completed -
max_delete_window
: maximum number of rows that will be deleted in a single call to the cleanup routine. This provides an upper limit on the time that the database could potentially remain blocked during that call. -
pruning_period
: minimum time between pruning passes
max_job_age
and pruning_period
can be specified in weeks, days, hours and those values can be combined. For example, it could be possible to specify a range of 1 week, 6 days, 12 hours. (This maps directly to the arguments taken by datetime.timedelta
, thus making parsing trivial.)
Changes proposed in this merge request:
-
parser.py
: extendSQLSchedulerConfig
with new options -
SQLDataStore.__init__()
: take two new optional parameters
Validation
These new options are not required, so BuildGrid starts with any valid config.
Adding these options to the default config also works:
schedulers:
- !sql-scheduler &state-database
storage: *cas-storage
connection-string: sqlite:///./example.db
automigrate: yes
connection-timeout: 15
poll-interval: 0.5
job-max-age: # [float type] Also supported: weeks, hours
days: 90
pruning-period:
hours: 48
max-delete-window: 10000
max-delete-window
,job-max-age
, and pruning-period
, must be either all None or all non-None to succeed validation.
Also setting all the values in job-max-age
or pruning-period
equal to 0 will fail validation.
Issues addressed
Related to #367 (closed)