Repair concurrent file uploads
Before the move to .Net Core, the import allowed simultaneous import requests for data concerning different DSDs. This feature was lost during the migration to .Net Core.
The purpose of this ticket is to bring back the original feature, that is: during a data import/transfer request, the transfer-service can process concurrently a max number (given by configuration, e.g. default: 10) of requests, as long as each request is targeting a different DSD. When the max number is exceeded then the system should queue the new request and start executing it only after all preceding requests in the queue are processed and once a processing slot becomes available.
Requests for data for the same DSD continue to be queued.
Technical implementation (for the documentation)
- when a new import is submitted for a DSD for which an import is currently already going on:
- Step 1 the request is added to a queue
- Step 2 the request is dequeued
- Step 3 The system detects that there is already an ongoing transaction for the same DSD and returns with the concurrent error msg.
- Step 4 the failed request is marked as completed unsuccsessfully.
- when a new import is submitted for a DSD for which no current import is currently going on but the max number of parallel imports is already reached:
- Step 1 the request is added to a queue
- Step 2 the request is dequed once there is free space for it to be threated (a parallel task is finished and takes a new request to process)
- Step 3 The system detects that there is NOT ongoing transaction for the same DSD and flags the transaction in progress.
- Step 4 the request is processed.
The differences with the specs of the future ticket #125 are:
- The queue is not persisted.
- all submited (queued) requests will be lost in a system failure
- all ongoing transactions are marked as canceled
- all queued requests are marked as canceled
- the system will not restart the queue where it left, when it becomes alive.
- each transfer-service manages its own inmemory queue, and is agnostic of the state of the queue of other instances targeting the same dataspace.
- all submited (queued) requests will be lost in a system failure
- When a request submited to the queue for a DSD with an ongoing transaction, the system deques it in the first slot available of the max number of parallel tasks. Meaning that it does not wait for the ongoing transaction for the same DSD to finish.