Skip to content

Reduce db size

Juliane Schmachtenberg requested to merge reduce_DB_size into master

Identification files with many samples and sample_groups exceeded 15 GB of RAM when DB was created. The size of the SQL DB and pandas df used to create the SQL DB is greatly reduced by not inserting null quantification values. SQL DB creation directly from df_standard, removing additional dict creation. When creating the quant file, the missing values must be re-inserted. When creating summary: for samples without spectra '-' are added, replacement of '-' values from map results by null values. Adapted test SQL files to new sql DB structure, added new tests test_full_with_spectra, test_quant_file_creation

Moved quant file creation from rules to extra function. Added insertion of 0 quant values. Restructuring into functions. Added test for quant file creation.

Summary writing for large files consumes too much RAM, now block wise reading for maps, quant and lca files.

Edited by Juliane Schmachtenberg

Merge request reports