Skip to content

`SQLDataStore`: add configuration parameters for automatic job pruning

Santiago Gil requested to merge santigl/scheduler-cleanup/cleanup-config into master

Before raising this MR, consider whether the following are required, and complete if so:

  • [ ] Unit tests
  • [ ] Metrics
  • Documentation update(s) TODO

If not required, please explain in brief why not.

Description

This extends the schema for sql-scheduler with options needed for the cleanup mechanism (#367 (closed)). That is:

  • max_job_age: maximum time that a job should be kept in the database after it was marked as completed
  • max_delete_window: maximum number of rows that will be deleted in a single call to the cleanup routine. This provides an upper limit on the time that the database could potentially remain blocked during that call.
  • pruning_period: minimum time between pruning passes

max_job_age and pruning_period can be specified in weeks, days, hours and those values can be combined. For example, it could be possible to specify a range of 1 week, 6 days, 12 hours. (This maps directly to the arguments taken by datetime.timedelta, thus making parsing trivial.)

Changes proposed in this merge request:

  • parser.py: extend SQLSchedulerConfig with new options
  • SQLDataStore.__init__(): take two new optional parameters

Validation

These new options are not required, so BuildGrid starts with any valid config.

Adding these options to the default config also works:

schedulers:
      - !sql-scheduler &state-database
        storage: *cas-storage
        connection-string: sqlite:///./example.db
        automigrate: yes
        connection-timeout: 15
        poll-interval: 0.5
        job-max-age: # [float type] Also supported: weeks, hours
          days: 90
        pruning-period:
          hours: 48
        max-delete-window: 10000

max-delete-window,job-max-age, and pruning-period, must be either all None or all non-None to succeed validation.

Also setting all the values in job-max-age or pruning-period equal to 0 will fail validation.

Issues addressed

Related to #367 (closed)

Edited by Santiago Gil

Merge request reports