## Moved to https://gitlab.com/groups/gitlab-org/-/epics/4181 GitLab.com's Postgres database is growing and we've reached about 5.8 TB as of March 2020. The growth rate is about 500 GB per 30 days (in March). This epic is meant to facilitate coordinating and identifying approaches to achieve two points: 1. Reduce the total size (in many cases a one-time win) 1. Reduce the database growth rate The epic is also meant to act as a source for explaining sources of database growth. Per-table statistics (physical size, row count estimates) can be found in this [sheet](https://docs.google.com/spreadsheets/d/1hOP62fZJFJdOyC1dzmT4DeF4xU2qsm3TzReHIquMMZI/edit#gid=1224268488). Database storage growth over time can be observed in [Grafana](https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1&fullscreen&panelId=10&from=now-30d&to=now). --- | Issue | Expected size reduction | Expected growth reduction | | --- | ------ |---------:| | Extract merge request diff BLOBs https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7356 | 1,700 GB | 60 GB / month | | Regular index bloat maintenance https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9563 | 500 GB | 120 GB / month | | 1-year CI retention strategy https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/5588 https://gitlab.com/gitlab-org/gitlab/-/issues/215646 | [minimum of 160 GB](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/5588#note_328476718) | tbd | | 3-months CI retention strategy https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/5588 https://gitlab.com/gitlab-org/gitlab/-/issues/215646 | [minimum of 320 GB](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/5588#note_328476718) | tbd | | Detect and remove unused / unneeded indexes https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/14 | tbd - indexes make up 40% of total db size | tbd | | Retention strategy for `ci_build_trace_sections` - https://gitlab.com/gitlab-org/gitlab/-/issues/32565 | TBD | 20GB / month | | Retention strategy for `audit events` - https://gitlab.com/gitlab-org/gitlab/-/issues/216847 | TBD | TBD | | Implement worker that `hard-delete` old CI jobs metadata - https://gitlab.com/gitlab-org/gitlab/-/issues/215646 | | | | Reduce the size of table `merge_request_diff_commits` - https://gitlab.com/gitlab-org/gitlab/-/issues/30440 | | |