orchestrator package design

Key Points

Job scheduling and management (manages jobs on other DMSs). One of the key packages of DMS, that is responsible for job orchestration on the platform level.

The general goal with the orchestrator is to create a module that’s able to manage jobs running on a remote machine. The job of the orchestrator is to keep track of all jobs, check their health, make deployment decisions such as the machine to use, amount of resources needed etc.

Work Package Deliverables

No	Deliverable	Proof
1	orchestrator package designed and specified	design specifications and tests written
2	breaking to implementation issues	issues for implementation defined and described, added to implementation package

Timeline

Related to DMS architecture package both in the sense that it depends on the definition of general architecture, however general architecture decisions are related to possibilities of design that will be discovered during package design.

Additional notes and context

1. Can be related to functionalities implemented in bacalhau packages pkg/bidstrategy pkg/orchestrator (see comment), pkg/requester (to be checked when designing implementation of dms.orchestrator package);
2. The design of NuNet job orchestration semantics, based on the outputs of the research done during 2023 Q4 - 2024 Q1: https://nunet.gitlab.io/research/blog/ (research blog, maintained by research teams);
3. See also initial job orchestration discussion document https://docs.google.com/document/d/1ERVkdOr0Hc5pfStqCElh-dSEa7E0-uX8JXhZcPxvJMU/edit