### Release Notes In milestone 14.8 we will release the ability for users to manually export their product usage data and manually upload it to GitLab. See [docs]() for more information. Noting that the features in the 14.8 release will be under a feature flag until we release an improvement in performance via https://gitlab.com/gitlab-org/product-intelligence/-/issues/513.  --- # **Proposal for Offline Usage Data Reporting MVC** _copied from [proposal](https://gitlab.com/gitlab-org/growth/product-intelligence/proposals/-/blob/master/doc/proposal_review_and_export_payload.md)_ ## **Summary** To improve acceptance of usage metrics we should allow users to review and export metrics that are being collected by Gitlab. ## **Requirements** * User can view the data we are collecting * User can provide the data to us outside of the GitLab application ## **Proposal** #### **Work Plan** The idea of this MVC is to move ServicePing payload preview (Admin - Settings - Metrics and Profiling - Usage statistics) to a new page within the Admin panel. We would need to implement export data to a JSON file. This file can be upload directly to Version application using a new ServicePing payload upload form. #### **Current State** Currently we have a ServicePing payload preview section (Admin - Settings - Metrics and Profiling - Usage statistics). Users are able to obtain a preview of all collected metrics and their values after clicking the “Preview Payload” button. Preview is not generated instantly, it takes a few seconds or longer. This has to be considered when designing a new page layout. The result is cached for two weeks https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/usage_data.rb#L39 but it’s possible to override the caching strategy. Two weeks caching means Preview would use payload generated by ServicePing most of the time. ##### Technical data Preview generation method: https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/admin/application_settings_controller.rb#L59 - We already have a json format, which can be reused for exporting data. Application setting \*\*application_setting\[version_check_enabled\] \*\* will have to be moved to the new controller. #### **Implementation Plan** 1. Create a new page Admin - Settings - Service Usage Data. This page will be under a Feature Flag 1. Move the [existing UI element](https://gitlab.com/gitlab-org/gitlab/blob/c71629429d9d43517d14ea4c748600148f1fb4a4/app/assets/javascripts/pages/admin/application_settings/payload_previewer.js#L22) (“Preview Payload” button and related text area) to the new page. Rename the button to “View Payload” to "Download Service Ping". UI and interaction is implemented in plain JS (not Vue component). We should keep the existing behaviour (not to load payload with the page to reduce unnnessesary and potentially heavy server request). 1. Reroute the GET /usage_data.html request to ApplicationSettingsController. I would suggest to leave moving this method to a new controller in future iterations when we have better idea regards the new page functionality. 1. Add a new button “Export Payload” as json. It will send GET /usage_data.json for data export. 1. Add a new page to Version app with a simple form. The form will have a file input and a submit button. 1. On the backend side create a controller/action to accept and validate the payoad file. Some basic validtion is essential to prevent service abuse. Also worth discussing possibility to add Captcha or something similar to add extra service abuse prevention. ## **Open questions** 1. The question [was raised ](https://gitlab.com/gitlab-org/gitlab/-/issues/346372#note_742948931)regarding the possibility to prevent exported data being modified. One of the solutions could be adding a checksum field to the exported json. However this is quite a complex solution. I would suggest moving it for later iterations ## **Potential concerns** 1. Caching.Preview generation might take time and better make sure we can execute it during a HTTP request (and not to cause a timeout) if cache is missing. ### **Estimation** I would suggest splitting the work into two issues. 1. Moving Payload Preview to a new page. This story is straightforward and I would estimate it as 3pts. 1. Implement Service ping upload and validation. 3pts.