The Transfer-service should mark all aborted transactions as closed & (timely) send email notifications
Issue: Error log information returned too late through status request and email notification is not received
For an data upload with 50 million of observations that stopped after 20 minutes, the user did not receive an email notification, and the explanatory message of the executionStatus ("TimeOutAborted") became visible in the transfer status request only after the user launched a second upload.
Background info:
The transer-service relies on the config parameter DataImportTimeOutInMinutes to know if a transaction should be considered as timed out, when it did not finished properly and it hasn't been marked as completed. When the value of DataImportTimeOutInMinutes is bigger than the time span elapsed since an aborted import started, the transaction can still be considered as in progress, even though there is no ongoing processing related to it.
Solution ideas:
-
Check if the transfer service could properly mark all current jobs as 'ended' (and send related e-mail notification messages) when asked to shut down (Is there a "OnShutdown" event?) -
Check if the transfer service could properly mark all previously cancelled jobs as 'ended' when starting up (done within the "OnStart" event?) This was decided to be done using the service bus. -
Clarify the TimedOutAborted
status. Change toTimedOut&Aborted
and use it only if the SQL queries or job really timed out and the jobs closed normally. "Killed" or something similar could be used if the jobs where killed externally. -
Review and improve the code to ensure that simply timed-out imports correctly send the email notifications at the end. -
Add check so that when doing a maintenance transactions doesnt occure at the same time as a transfer transaction, more details in link
Additional bug fix
with the following data files (EDD) : IDD.xlsx IDD.xml and structure file OECD.SOC-SDD_IDD-1.0-all.xml
When I upload these in qa:reset
, I receive the following email in which the summary part is complete:
But when I try the method /{version}/status/request
with swagger, I get an empty summary part (except for the destinationspace
):
- The issue happens when the import requests fails before the transaction gets recorded.