Add compute minute data to API

Ben Kingrequested to merge
benjaminking-api-minutes-usage into master

What does this MR do and why?

Addresses: #12379

Summary

This change adds CI (Continuous Integration) minutes usage information to the namespace API response. It provides a "breakdown" of usage from both the monthly quota, and any assigned or purchased compute minutes.

This is helpful for:

  • Users (personal namespace) and Group Owners (group namespaces) to track usage at a glance, including understanding when they are about to or have gone over their monthly quota.
  • For Support team members that need to check personal namespace compute minutes usage breakdowns for troubleshooting. A user's quota breakdown can only be visibly checked if impersonating the user's account.

User Namespace Example

Image/Text Description 
{
  "id": 125,
  "name": "testuser1",
  "path": "testuser1",
  "kind": "user",
  "full_path": "testuser1",
  "parent_id": null,
  "avatar_url": "https://www.gravatar.com/avatar/f5f234e6b4028f68a41cf0a3491af8c8179f851d3906cd8bca8d63c91d831a59?s=80&d=identicon",
  "web_url": "http://gdk.local:3000/testuser1",
  "shared_runners_minutes_limit": 1,
  "ci_minutes_usage": {
    "total_minutes_used": 2,
    "monthly_minutes_used": 1,
    "purchased_minutes_used": 1
  },
  "extra_shared_runners_minutes_limit": 10,
  "additional_purchased_storage_size": 0,
  "additional_purchased_storage_ends_on": null,
  "billable_members_count": 1,
  "seats_in_use": 1,
  "max_seats_used": 0,
  "max_seats_used_changed_at": null,
  "end_date": null,
  "plan": "free",
  "trial_ends_on": null,
  "trial": false
}
 (As ADMIN) - Admin can verify details regarding the user's namespace including breakdown of compute minutes usage. 
{
  "id": 125,
  "name": "testuser1",
  "path": "testuser1",
  "kind": "user",
  "full_path": "testuser1",
  "parent_id": null,
  "avatar_url": "https://www.gravatar.com/avatar/f5f234e6b4028f68a41cf0a3491af8c8179f851d3906cd8bca8d63c91d831a59?s=80&d=identicon",
  "web_url": "http://gdk.local:3000/testuser1",
  "ci_minutes_usage": {
    "total_minutes_used": 2,
    "monthly_minutes_used": 1,
    "purchased_minutes_used": 1
  },
  "billable_members_count": 1,
  "seats_in_use": 1,
  "max_seats_used": 0,
  "max_seats_used_changed_at": null,
  "end_date": null,
  "plan": "free",
  "trial_ends_on": null,
  "trial": false
}

(As USER) - The user can see the new ci_minutes_usage object.

This matches what the user can already see in the UI at: http://localhost:3000/-/profile/usage_quotas#pipelines-quota-tab

image.png

Group Namespace Example

The same concept applies for groups:

/

Image/Text Description 
{
  "id": 35,
  "name": "Twitter",
  "path": "twitter",
  "kind": "group",
  "full_path": "twitter",
  "parent_id": null,
  "avatar_url": null,
  "web_url": "http://gdk.local:3000/groups/twitter",
  "members_count_with_descendants": 6,
  "root_repository_size": 2184146,
  "projects_count": 3,
  "shared_runners_minutes_limit": 400,
  "ci_minutes_usage": {
    "total_minutes_used": 2,
    "monthly_minutes_used": 2,
    "purchased_minutes_used": 0
  },
  "extra_shared_runners_minutes_limit": null,
  "additional_purchased_storage_size": 0,
  "additional_purchased_storage_ends_on": null,
  "billable_members_count": 12,
  "seats_in_use": 12,
  "max_seats_used": 0,
  "max_seats_used_changed_at": null,
  "end_date": null,
  "plan": "free",
  "trial_ends_on": null,
  "trial": false
}
 (As ADMIN) - Can see all relevant details 
{
  "id": 35,
  "name": "Twitter",
  "path": "twitter",
  "kind": "group",
  "full_path": "twitter",
  "parent_id": null,
  "avatar_url": null,
  "web_url": "http://gdk.local:3000/groups/twitter",
  "members_count_with_descendants": 6,
  "root_repository_size": 2184146,
  "projects_count": 3,
  "ci_minutes_usage": {
    "total_minutes_used": 2,
    "monthly_minutes_used": 2,
    "purchased_minutes_used": 0
  },
  "billable_members_count": 12,
  "seats_in_use": 12,
  "max_seats_used": 0,
  "max_seats_used_changed_at": null,
  "end_date": null,
  "plan": "free",
  "trial_ends_on": null,
  "trial": false
}
 (As USER) - Can see the new object

Database

Full Database Query

SELECT namespace_id, SUM(amount_used) AS sum_amount_used
FROM ci_namespace_monthly_usages
WHERE date = DATE_TRUNC('month', NOW() AT TIME ZONE 'UTC')
  AND namespace_id IN (/* list of namespace IDs */)
GROUP BY namespace_id

Query Execution Plan

 GroupAggregate  (cost=0.57..47.22 rows=4 width=40) (actual time=33.704..47.262 rows=2 loops=1)
   Group Key: ci_namespace_monthly_usages.namespace_id
   Buffers: shared hit=52 read=55 dirtied=27
   WAL: records=27 fpi=27 bytes=151623
   I/O Timings: read=44.121 write=0.000
   ->  Index Scan using idx_ci_namespace_monthly_usages_namespace_id_date_shard_number on public.ci_namespace_monthly_usages  (cost=0.57..47.15 rows=4 width=11) (actual time=25.844..47.195 rows=20 loops=1)
         Index Cond: ((ci_namespace_monthly_usages.namespace_id = ANY ('{9970,278964,6543,42,101,2384,55123,900001,13,777,204,650819,1200,34567,88,5,999999,45231,73012,600}'::bigint[])) AND (ci_namespace_monthly_usages.date = date_trunc('month'::text, (now() AT TIME ZONE 'UTC'::text))))
         Buffers: shared hit=52 read=55 dirtied=27
         WAL: records=27 fpi=27 bytes=151623
         I/O Timings: read=44.121 write=0.000
Settings: work_mem = '100MB', random_page_cost = '1.5', effective_cache_size = '338688MB', jit = 'off', seq_page_cost = '4'
Query ID: -3577008500000148012

Summary:

  Time: 48.607 ms  
  - planning: 1.251 ms  
  - execution: 47.356 ms  
    - I/O read: 44.121 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 52 (~416.00 KiB) from the buffer pool  
  - reads: 55 (~440.00 KiB) from the OS file cache, including disk I/O  
  - dirtied: 27 (~216.00 KiB)  
  - writes: 0

How to set up and validate locally

  1. Configure either compute quota for all namespaces or compute quota for a top-level group, defining a low value such as 5 or 10. This defines the monthly_minutes_used value.
  2. Via Rails console:
# Define the namespace you set the compute quota for
namespace = Namespace.find_by_id(<id>)
namespace.extra_shared_runners_minutes_limit = 10
namespace.save!
  1. Verify via the UI in usage quotas for a group (such as via: http://gdk.local:3000/groups/twitter/-/usage_quotas#pipelines-quota-tab) that both the monthly quota, and "extra" quota are defined:

image.png

  1. Confirm the API functions as expected, but at this time returns 0 for all results in ci_minutes_usage:
...
"shared_runners_minutes_limit": 10,
  "ci_minutes_usage": {
    "total_minutes_used": 0,
    "monthly_minutes_used": 0,
    "purchased_minutes_used": 0
  },
"extra_shared_runners_minutes_limit": 10,
...
  1. Connect GitLab Runner, and install this as an Instance Runner. This is required to ensure the quota is utilised.
  2. Run some example jobs, preferably with sleep so that these take a few minutes.
  3. Confirm that, via both the UI and API:
    1. monthly_minutes_used is used first, which also updates total_minutes_used
    2. When the full quota of monthly minutes is used up, it will start to use purchased_minutes_used

image.png

...
"shared_runners_minutes_limit": 10,
  "ci_minutes_usage": {
    "total_minutes_used": 11,
    "monthly_minutes_used": 10,
    "purchased_minutes_used": 1
  },
"extra_shared_runners_minutes_limit": 10,
...

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Ben King

Merge request reports