Skip to content

Create table `project_data_transfers` to store egress data per project

What does this MR do and why?

Contributes to #383771

We want to store egress data per project in the database. The data will be aggregated per project per month.

git_egress represents tracked egress from git-related services in bytes. Later this table will be extended with new fields for other services.

The implementation is similar to ci_project_monthly_usages.

Changes

  1. Create table project_data_transfers
  2. Define Rails model Projects::DataTransfer
  3. Add factory and tests for the new table
  4. Document purpose and ownership of the new table

High-level description

We need a storage to keep and access egress data per project. This data will be aggregated by month.

All collected data for May 2022 for project 1 will be stored in project_data_transfer table.

project_id namespace_id git_egress date created_at
1 1 200 2022-05-01 2022-05-02 12:00:00
2 2 500 2022-05-01 2022-05-01 12:00:00
1 1 700 2022-06-01 2022-06-05 15:00:00

Later, this table will be extended with other egress fields.

We are going to keep this data for 12 months and delete it after this period of time.

Database

Click to expand

Up

> bin/rails db:migrate

main: == 20230117173433 CreateProjectDataTransfer: migrating ========================
main: -- transaction_open?()
main:    -> 0.0000s
main: -- create_table(:project_data_transfers, {})
main:    -> 0.0040s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- current_schema()
main:    -> 0.0002s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE project_data_transfers\nADD CONSTRAINT project_data_transfers_project_year_month_constraint\nCHECK ( (date = date_trunc('month', date)) )\nNOT VALID;\n")
main:    -> 0.0006s
main: -- current_schema()
main:    -> 0.0002s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- execute("ALTER TABLE project_data_transfers VALIDATE CONSTRAINT project_data_transfers_project_year_month_constraint;")
main:    -> 0.0003s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20230117173433 CreateProjectDataTransfer: migrated (0.1335s) ===============

Down

> bin/rails db:rollback:main

main: == 20230117173433 CreateProjectDataTransfer: reverting ========================
main: -- transaction_open?()
main:    -> 0.0000s
main: -- drop_table(:project_data_transfers)
main:    -> 0.0015s
main: == 20230117173433 CreateProjectDataTransfer: reverted (0.0390s) ===============

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasilii Iakliushin

Merge request reports