Skip to content

Resolve "Export table/view data to CSV/XML/JSON"

Nigel Gott requested to merge 323-export-table-view-data-to-csv into develop

Closes #323 (closed) Closes #325 (closed) Closes #326 (closed)

This merge request adds the ability to export views and tables to CSV files. It does so by introducing a new set of export endpoints on the backend and a export modal on the frontend. When the user starts an export in the backend a new export job is created and submitted onto an export job only celery queue. A new export only worker will then pickup export tasks, export the view/table and store it in a file in the default_storage. The frontend will poll every 1 second until the job completes and then display the download link to the file in the storage.

TODO:

  • Test export in cloudron
  • Test can open csv's in word
  • Test can't export json and xml staging
  • delete convertOptionsToJson

Testing notes / guide:

  • Old export files are cleaned up, set the EXPORT_FILE_DURATION_MINUTES django config setting to 5 to test this easier, look in the new dev.sh tab "backend export worker" and "backend beat worker" to see jobs being scheduled and jobs being cleaned up.
  • The fill table --add-columns command is great for testing + now makes more columns with better names for the columns (us date + eu date columns are now created etc)
  • Can export views with
    • Filters active, the resulting csv is filtered
    • Sorts active, the resulting csv is sorted
    • Columns ordered differently which show up in the csv in the same order
    • Rows manually ordered, which show up in the csv in the same manual order
  • Can export tables on their own without a view selected
  • Can rename views via the new view context menu
  • Long exports memory and speed performance has been optimized (see https://gitlab.com/bramw/baserow/-/snippets/2121706 for testing harness, results and different implementations)
  • The server crashing or returning any form of error causes the client to stop polling
  • Starting new jobs whilst old ones are running cancels the existing ones
  • Tested the various charset + include header + col separator options and they are respected
  • General test of core baserow features in staging also looks good with changes to the base saas image.
Edited by Bram Wiepjes

Merge request reports