False failures when uploading datasets
Summary: We are experiencing a series of failures when uploading data to our .Stat implementation. Please see the email, JSON and screenshot below.
Failed .Stat data request for 'STC_CSIPDF_VM_CMACA(1.0)' in 'design-cpt' (ID 4312).msg
We’ve divided our data into files that include 5 million rows each, which usually end up being around 500MB in size. They are being uploaded unzipped using the path method. As you can see from the logs above, our upload does not always fail. However, we believe that even if the .Stat error log identifies a failure, the data is still being inserted successfully. We think this is the case because when we count all the attempted rows entered and the availability count, they are equal. See the spreadsheet and screenshot below.
From our investigations it seems like the error is not causing a transaction rollback, and therefore giving us a what we call a “False Failure”. The only common variable we were able to identify across our failures was that they are utilizing one DSD for multiple data flows. However, these files do not always fail and even if they did we believe this to be a valid action.
Environment
- .Stat JS Yay
- .Stat Core Gingerbread
- Windows Azure Cloud
- Azure SQL Elastic Pool
Steps to reproduce
- Upload data file using path approach
- Data file is unzipped
- Data file includes 5 million rows
- Data file is around 500 MB in size
Expected Result
- Data entered into .Stat
- Successful log entry
Or
- Data NOT entered into .Stat
- Error log entry
Actual Result
- Data entered into .Stat. (probable)
- Error log entry
- No transaction rollback