### About The new [data retention guidelines](https://docs.gitlab.com/development/data_retention_policies/) are asking us to consider data retention lifecycles (essentially, can we delete, or move outside of PG, data as it gets old). The guidelines say: _Features should be designed with the understanding that data is not permanent_. Some of the context for this initiative is the risk to GitLab availability of the size of its data: * https://gitlab.com/gitlab-com/gl-security/product-security/data-security/data-security-team/-/issues/60+ * https://gitlab.com/groups/gitlab-org/-/epics/16520+ ### Problem To allow [User contribution mapping](https://docs.gitlab.com/user/project/import/#user-contribution-and-membership-mapping) track what imported data to reassign later, we store data in two tables: | Table | Model | Purpose | | ------ | ------ | ------ | | `import_source_user_placeholder_references` | `Import::SourceUserPlaceholderReference` | References to all imported data, except memberships | | `import_placeholder_memberships` | `Import::Placeholders::Membership` | Membership data https://gitlab.com/gitlab-org/gitlab/-/issues/477845 | The data in `import_source_user_placeholder_references` is quickly growing. The feature was released on GitLab.com in December 2024, but at time of writing, we're storing around 17GB of user mapping data on the primary, which inflates to around 290GB when including the read-only replicas ([source](https://gitlab-com.gitlab.io/gl-infra/platform/stage-groups-index/import.html)). (Data from 26 March 2025) | Table name | Table size | Database storage (w/ replicas) | |------------|------------|--------------------------------| | import_source_user_placeholder_references | 15.99 GB | 271.79 GB | | import_placeholder_memberships | 0.91 GB | 15.42 GB | This epic collects issues that will reduce the size of data in these tables. ### Opportunity We have a good opportunity to enact a Data Retention Policy for this data now before we reach the data retention period and delete any data. We can give customers a good amount of warning before the feature is affected. ### Proposed solution #### 1. Implement a reassign process that doesn't require placeholder references when contributions are linked to a placeholder user. This solution will already significantly reduce the size of the table, as most contributions fall into this category. #### 2. Implement a data retention policy. With solution 1, the `import_source_user_placeholder_references` should already decrease substantially; however, we will want to implement a data retention policy to avoid keeping old records and to clean up the `import_placeholder_memberships`, which will continue to be used even after solution 1. The user will have one year to complete the reassignment. After this period, contributions will be marked as `keep as placeholders`, and users will no longer be able to make changes. Additionally, records in `import_placeholder_memberships` and `import_source_user_placeholder_references` will be deleted. This solution will be implemeted in https://gitlab.com/groups/gitlab-org/-/epics/19762