GitLab

The following SqlBulkCopy error was reported in the dotstatsuite-core-transfer-user.log:

2019-10-10 10:57:06,395|[7]|codebase-install-test|33|ERROR|DOTSTATCOR-DEV1|dotstatsuite-core-transfer|DotStatServices.Transfer.Controllers.ImportController+<>c__DisplayClass10_0`1+<<Import>b__0>d.MoveNext|Column 'DIM_52' does not allow DBNull.Value.|System.InvalidOperationException: Column 'DIM_52' does not allow DBNull.Value.
   at System.Data.SqlClient.SqlBulkCopy.ConvertValue(Object value, _SqlMetaData metadata, Boolean isNull, Boolean& isSqlType, Boolean& coercedToDataFeed)
   at System.Data.SqlClient.SqlBulkCopy.ReadWriteColumnValueAsync(Int32 col)
   at System.Data.SqlClient.SqlBulkCopy.CopyColumnsAsync(Int32 col, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.CopyRowsAsync(Int32 rowsSoFar, Int32 totalRows, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken)
   at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServer(IDataReader reader)
   at DotStat.Db.Repository.SqlServer.SqlDataStoreRepository.BulkInsertData(IEnumerable`1 observations, CodeTranslator translator, Dataflow dataflow, Int32 maxErrorCount)
   at DotStat.Transfer.Consumer.SqlConsumer.Save(TransferParam transferParam, Dataflow dataflow, TransferContent transferContent, Int32 maxErrorCount) in C:\GitLab\dotstatsuite-core-transfer\DotStat.Transfer\Consumer\SqlConsumer.cs:line 158
   at DotStat.Transfer.Manager.SdmxFileToSqlTransferManager.Transfer(SdmxFileToSqlTransferParam transferParam) in C:\GitLab\dotstatsuite-core-transfer\DotStat.Transfer\Manager\SdmxFileToSqlTransferManager.cs:line 21
   at DotStatServices.Transfer.Controllers.ImportController.<>c__DisplayClass10_0`1.<<Import>b__0>d.MoveNext() in C:\GitLab\dotstatsuite-core-transfer\DotStatServices.Transfer\Controllers\ImportController.cs:line 206
|(null)|(null)|(null)|(null)

Such an error should not happen, because all data input is validated before loading it into SQL, right?

Improve observations validation process; the data reader should take an optimistic approach, after the first validation error, the validator will iterate through the rest of the observations to detect more errors until it reaches the end of the file or the maximum number allowed errors, before the transaction is cancel.