Resolve "Export table/view data to CSV/XML/JSON"
Closes #323 (closed) Closes #325 (closed) Closes #326 (closed)
This merge request adds the ability to export views and tables to CSV files. It does so by introducing a new set of export endpoints on the backend and a export modal on the frontend. When the user starts an export in the backend a new export job is created and submitted onto an export job only celery queue. A new export only worker will then pickup export tasks, export the view/table and store it in a file in the default_storage. The frontend will poll every 1 second until the job completes and then display the download link to the file in the storage.
TODO:
-
Test export in cloudron -
Test can open csv's in word -
Test can't export json and xml staging -
delete convertOptionsToJson
Testing notes / guide:
- Old export files are cleaned up, set the EXPORT_FILE_DURATION_MINUTES django config setting to 5 to test this easier, look in the new dev.sh tab "backend export worker" and "backend beat worker" to see jobs being scheduled and jobs being cleaned up.
- The fill table --add-columns command is great for testing + now makes more columns with better names for the columns (us date + eu date columns are now created etc)
- Can export views with
- Filters active, the resulting csv is filtered
- Sorts active, the resulting csv is sorted
- Columns ordered differently which show up in the csv in the same order
- Rows manually ordered, which show up in the csv in the same manual order
- Can export tables on their own without a view selected
- Can rename views via the new view context menu
- Long exports memory and speed performance has been optimized (see https://gitlab.com/bramw/baserow/-/snippets/2121706 for testing harness, results and different implementations)
- The server crashing or returning any form of error causes the client to stop polling
- Starting new jobs whilst old ones are running cancels the existing ones
- Tested the various charset + include header + col separator options and they are respected
- General test of core baserow features in staging also looks good with changes to the base saas image.
Edited by Bram Wiepjes