Skip to content

storage package design

Key Points

Design and specification of an interface to file storage and retrieval. There are cases where the DMS itself or jobs run by the DMS need files to work with. In order to supply this demand, this package will be responsible for the following for interfacing with multiple storage providers (including locals disk) as well as transferring files between DMSs themselves;

Work Package Deliverables

Design the functionalities and provide specifications for implementation package on the following items, including priorities and scope of each item to be implemented within this milesone.

No Deliverable Proof
1 Serving for an interface from DMS to multiple types of storage mechanisms. Storage abstraction layer on DMS which allows to access different storage providers via unified interface Designed and specified
2 Implementation of chosen set of storage mechanism using the interface Designed and specified
3 Allow DMSs to transfer files between each other. This can only be achieved using the network module. Hence, this package will only be responsible for reading and writing the file from and to the access type based on the list above, doing error checks on the transfer and seeking to a specific location. The network module will take care of the rest. Designed and specified
4 Disk storage management on each DMS for data related to DMS and jobs deployed by DMS and adapter to external storage services; Designed and specified
5 Encryption at rest option (self provided key instead of default encryption provided by the system for privacy) Designed and specified

Additional notes and context

  1. Potential storage providers to be designed within the scope of this work package:
    1. Local storage
      • docker volumes
      • local disks
    2. HTTP and FTP
    3. IPFS
    4. Google Cloud
    5. AWS S3 Bucket etc…
    6. alternative to gitlab for image registry (eg: decentralized image registry);
  2. Check the following bacalhau packages for ideas / implementations or related functionalities pkg/ipfs, pkg/s3, pkg/storage
Edited by kabir