SQL-backed scheduler should delete completed jobs from database after a while

Context

When using an SQL-backed scheduler, a Job that is completed will remain in the database for ever. This means that the database size grows without limit.

The scheduler should be able to automatically perform a cleanup of records that are safe to delete. That definition of safety will be dictated by the guarantees that the List/GetOperations() API must provide.

Task Description

• Implement a routine to efficiently delete completed jobs older than a date from the database in an efficient manner
• SQLDataStore: Trigger the mechanism that automatically removes jobs older than a given range
• Extend settings/schemas/scheduler/sql.yaml with necessary configuration options. For example: maximum job age, cleanup size window (for reducing time DB is blocked)
• Document new feature, update sample config files
• Document process of enabling the feature for already-populated DBs (preparation step required? separate script to perform a manual cleanup?)

Acceptance Criteria

The SQL scheduler can be configured to automatically remove jobs after a specific amount of time after their completion. When that feature is enabled, the database will only contain records that are inside that time window.

In addition, the cleanup operation does not impact severely the performance of the scheduler.

Consider whether the following are required, and complete if so:

• Unit tests
• Metrics
• Documentation update(s)