PhiPhi: fix the max size of the gather batch row for MB
Also refactored the manual upload so that it uses the gather batches (this commit) is after the refactor of the code.
It worked like a charm! I did a huge gather and I got this from the analysis of the size of each row for the gather:
| row_count | total_mb | avg_row_mb | stddev_row_mb | min_row_mb | max_row_mb | p25_mb | median_mb | p75_mb | p90_mb | p95_mb | p99_mb | coefficient_of_variation |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 44 | 211.49 | 4.8067 | 0.8731 | 0.053 | 5.0 | 4.9836 | 4.9908 | 4.9964 | 4.9994 | 4.9998 | 4.9999 | 0.182 |