Skip to content

Stop duplicating state in Scheduler and DataStore

Adam Coldrick requested to merge sotk/stateless-scheduler into master

Description

At the moment, we have two copies of the state when a data store is configured. There's no need for that and it hinders scalability, so this MR refactors Scheduler to use new DataStore methods for any stateful logic and extends SQLDataStore to implement these methods when necessary. The old scheduler code is also refactored into a new implementation of DataStoreInterface, called MemoryDataStore. This can be used for testing, or when you don't want to set up a database for whatever reason, however it doesn't support surviving restarts and will never be able to work in the scalable BuildGrid architecture.

This MR also fixes a couple of bugs which were brought to light by doing this work.

Changes proposed in this merge request:

  • Make Scheduler use DataStore for all interactions with job/operation/lease state.
  • Extend SQLDataStore to support that
  • Add an in-memory implementation of DataStoreInterface
  • Fix state persistence of cancelled jobs
  • Fix persistence of "done" status of jobs
  • Stop treating jobs' platform requirements as a dictionary in the ORM models
  • Add tests

This merge request, when merged, will address issue/bug:

#184 (closed)

Edited by Adam Coldrick

Merge request reports