Re-import projects into project_build_artifacts_size_refreshes

Production Change

Change Summary

This is similar to what we did on #7215 (closed).

This is needed to be executed once gitlab-org/gitlab#368801 (closed) is released on production.

We want to refresh the build artifacts size stats of the projects with inaccurate values. Caused by gitlab-org/gitlab#368326 (closed).

Given it's hard to find specific projects that stats have just most recently been inaccurate, we would like to run this refresh task again on the list of affected projects. Then we would closely monitor once a project ends up with negative storage stats again. This way it would be easier for us to find which projects to trace counter increments and flushes in the logs.

We want to reimport the projects from the remote CSV (updated with project IDs that still have inaccurate stats as of August 17, 2022).

We need someone with write access to prod DB to run some lines on the Rails console to execute the rake task. Once the re-import is done, the background worker will eventually pick up the refresh work.

This would be running the same lines of code as hhttps://gitlab.com/gitlab-com/gl-infra/production/-/issues/7215, just the CSV URL will change.

Change Details

  1. Services Impacted - ServicePostgres
  2. Change Technician - @iamricecake
  3. Change Reviewer - TBD
  4. Time tracking - 55m
  5. Downtime Component - No downtime

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - N/A

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - around 35 minutes

  • On the rails console, run: 
    • > def measure_duration
*   start_time = Time.now
*   yield
*   duration = Time.now - start_time
*   puts "Done in #{duration}"
*   duration
* end
> Rails.application.load_tasks
> csv_url='https://storage.googleapis.com/verify_pipeline_insights/affectedprojectids_2022_8_17.csv'
> measure_duration { Rake::Task['gitlab:refresh_project_statistics_build_artifacts_size'].invoke(csv_url) }
    • Take note of the output of the rake task, showing the number of imported projects.
    • The CSV file contains around 809K rows of project IDs.

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - less than 1 minute

  • Confirm records are created in the new project_build_artifacts_size_refreshes table.
    • In the Rails console, we can run: 
      Projects::BuildArtifactsSizeRefresh.remaining.count

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - less than 1 minute

  • Truncate the project_build_artifacts_size_refreshes table
    • In the Rails console, we can run: 
      Projects::BuildArtifactsSizeRefresh.in_batches { |r| r.delete_all; sleep(0.5) }

Monitoring

Keep an eye on the size of new table to see progress:

Thanos

Look at the Patroni dashboard: https://dashboards.gitlab.net/d/patroni-main/patroni-overview?orgId=1

Change Reviewer checklist

C4 C3 C2 C1:

  • The scheduled day and time of execution of the change is appropriate.
  • The change plan is technically accurate.
  • The change plan includes estimated timing values based on previous testing.
  • The change plan includes a viable rollback plan.
  • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
  • The change plan includes success measures for all steps/milestones during the execution.
  • The change adequately minimizes risk within the environment/service.
  • The performance implications of executing the change are well-understood and documented.
  • The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
  • The change has a primary and secondary SRE with knowledge of the details available during the change window.

Change Technician checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents.
  • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Bob Van Landuyt