Skip to content

Ensure namespace/project used minutes are reset

Fabio Pitino requested to merge ensure-minutes-reset-for-all-namespaces into master

What does this MR do?

Fixes: https://gitlab.com/gitlab-org/gitlab-ce/issues/66679

After fixing https://gitlab.com/gitlab-org/gitlab-ce/issues/65540 we are experiencing still some customers complaining that some statistics are wrong. In attempt to fix the issue previously we have nested the minutes usage stats reset under the query Namespace.with_ci_minutes_notification_sent to reduce the queries, given that we send notifications for namespaces that have used CI minutes.

I'm not sure whether this is the fix for what the customer are complaining in the issue but the feeling is that we have recalculated the extra minutes limit (purchase minutes) but not reset the actual minutes used to 0.

With this change I'm forcing to reset the statistics for all namespaces. This means the worker will run for longer and more queries will be performed but we don't depend on other factors such as with_ci_minutes_notification_sent.

Previously in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/15002/diffs?view=parallel#diff-content-4751ff975b3b4fa3c112de746d198f1360a3965a we were resetting CI minutes used where.not(shared_runners_seconds: 0) but that caused the query to timeout with ProjectStatistics and left customers with wrong stats between namespace and project. That MR (https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/15002) attempted to fix this problem but it might have introduced an issue where "if we have not sent the CI minutes notification" for the namespace usage we would not reset the minutes used.

After deployment steps

To fix the statistics we need to first force reset the namespace statistics (before September 1st) and project statistics for all otherwise the recalculation of namespaces.extra_shared_runners_minutes_limit for the next month will be wrong because not all namespace_statistics.shared_runners_seconds were reset last time. That would cause customers to lose purchased minutes again.

The following snippet should be run on production Rails console prior to September 1st (before running ClearSharedRunnersMinutesWorker)

# ensure shared runners seconds are reset to 0 for projects and namespaces. We missed this on the previous month.
Namespace.select(:id).each_batch do |namespaces|
  Namespace.transaction do
    NamespaceStatistics.where(namespace: namespaces).where.not(shared_runners_seconds: 0).update_all(
      shared_runners_seconds: 0, shared_runners_seconds_last_reset: Time.now)
    ProjectStatistics.where(namespace: namespaces).where.not(shared_runners_seconds: 0).update_all(
      shared_runners_seconds: 0, shared_runners_seconds_last_reset: Time.now)
  end
end

# run the worker again to fix the statistics. Prior to September 1st.
ClearSharedRunnersMinutesWorker.new.perform

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Performance and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Fabio Pitino

Merge request reports