This project is archived. Its data is read-only.

Add ability to export all data for a given user

Add ability to export all data for a given user.

This could be an "Export all user data" option in the user settings dropdown,

This will open a modal with different buttons to export data, user data, messages, etc. They will be separate to lessen the burden on the server (no need to zip into a single file and have everything in memory).

We will probably use ndjson to more easily stream out the data so we don't have to put it all in memory.

This action will be rate limited to every 12 hours(undecided) to lessen the burden on the servers.

Data associated with a user

  • User data user-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1950
    • Do we censor GitHub tokens on export?
      • Not including tokens in the serialized user object
  • Identities from identity-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1951
  • User settings from user-settings-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1959
  • Group favorites user-group-favourites-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1960
  • Groups (based on whether it is based on their own user sd.type = GH_USER_SAME or one of the sd.extraAdmin) -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1962
  • Rooms troupe-schema (based on membership troupe-user-schema) (also based on whether it is based on their own user sd.type = GH_USER_SAME or one of the sd.extraAdmin) -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1967
  • favorites from user-troupe-favourites-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1966
  • last access data from user-troupe-last-access-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1969
  • Messages chat-message-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1941
    • Not necessarily connected to room membership because you can leave a room
  • chat-message-backup-schema
    • This was added because we have reports delete messages and we want to be able to restore if something was falsely reported. I'm not sure whether to include messages from this
  • Invites troupe-invite-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1971
  • Removed user from a room troupe-removed-user-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1973
  • Push notification settings push-notification-device-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1974
  • URI lookup uri-lookup-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1975
  • Legacy billing subscriptions subscription-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1976
  • Known external access known-external-access-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1977
  • Fingerprints fingerprint-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1978
  • OAuth clients oauth-client-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1979
  • OAuth codes oauth-code-schema
    • Do we censor codes on export?
    • I don't see a use case or reason to export this
  • OAuth access tokens oauth-access-token-schema -> https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1980
    • Do we censor tokens on export?
  • Topics -> Topics was deprecated and had export available before: https://gitlab.com/gitlab-org/gitter/webapp/-/merge_requests/1199
    • topic-schema
    • reply-schema
    • forum-reaction-schema
    • forum-subscription-schema
    • forum-notification-schema

TODO before GA

  • Reduce rate-limit to once per 3 hours, server/api-web/export/generate-export-resource.js#L16-17
  • Remove staff only restriction, server/api-web/export/generate-export-resource.js#L58-62
  • Put export behind config so we can disable it

Dev notes

https://github.com/agenda/agenda

Edited Aug 21, 2020 by Eric Eastwood
Assignee Loading
Time tracking Loading