Skip to content
Snippets Groups Projects
Verified Commit e10ed189 authored by Rajendra Kadam's avatar Rajendra Kadam :two: Committed by GitLab
Browse files

Add GraphQL API for visualizing dedicated hosted runner compute usage

Add filters on billing_month and year

Allows instance admins on GitLab Dedicated to pull the data using the API

Changelog: added
EE: true
MR: !179854
parent a3829b9a
No related branches found
No related tags found
2 merge requests!180727Resolve "Extend job archival mechanism to the whole pipeline",!179854Add GQL API to fetch dedicated hosted runner usage
Showing
with 514 additions and 0 deletions
......@@ -350,6 +350,24 @@ Returns [`CiConfig`](#ciconfig).
| <a id="queryciconfigsha"></a>`sha` | [`String`](#string) | Sha for the pipeline. |
| <a id="queryciconfigskipverifyprojectsha"></a>`skipVerifyProjectSha` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in GitLab 16.5. **Status**: Experiment. If the provided `sha` is found in the project's repository but is not associated with a Git reference (a detached commit), the verification fails and a validation error is returned. Otherwise, verification passes, even if the `sha` is invalid. Set to `true` to skip this verification process. |
 
### `Query.ciDedicatedHostedRunnerUsage`
Compute usage data for runners across namespaces on GitLab Dedicated. Defaults to the current year if no year or billing month is specified. Ultimate only.
Returns [`CiDedicatedHostedRunnerUsageConnection`](#cidedicatedhostedrunnerusageconnection).
This field returns a [connection](#connections). It accepts the
four standard [pagination arguments](#pagination-arguments):
`before: String`, `after: String`, `first: Int`, and `last: Int`.
#### Arguments
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="querycidedicatedhostedrunnerusagebillingmonth"></a>`billingMonth` | [`Date`](#date) | First day of the month to retrieve data for. |
| <a id="querycidedicatedhostedrunnerusagegrouping"></a>`grouping` | [`GroupingEnum`](#groupingenum) | Groups usage data by instance aggregate or root namespace. |
| <a id="querycidedicatedhostedrunnerusageyear"></a>`year` | [`Int`](#int) | Year to retrieve data for. |
### `Query.ciMinutesUsage`
 
Compute usage data for a namespace.
......@@ -13220,6 +13238,29 @@ The edge type for [`CiConfigStage`](#ciconfigstage).
| <a id="ciconfigstageedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. |
| <a id="ciconfigstageedgenode"></a>`node` | [`CiConfigStage`](#ciconfigstage) | The item at the end of the edge. |
 
#### `CiDedicatedHostedRunnerUsageConnection`
The connection type for [`CiDedicatedHostedRunnerUsage`](#cidedicatedhostedrunnerusage).
##### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="cidedicatedhostedrunnerusageconnectionedges"></a>`edges` | [`[CiDedicatedHostedRunnerUsageEdge]`](#cidedicatedhostedrunnerusageedge) | A list of edges. |
| <a id="cidedicatedhostedrunnerusageconnectionnodes"></a>`nodes` | [`[CiDedicatedHostedRunnerUsage]`](#cidedicatedhostedrunnerusage) | A list of nodes. |
| <a id="cidedicatedhostedrunnerusageconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. |
#### `CiDedicatedHostedRunnerUsageEdge`
The edge type for [`CiDedicatedHostedRunnerUsage`](#cidedicatedhostedrunnerusage).
##### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="cidedicatedhostedrunnerusageedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. |
| <a id="cidedicatedhostedrunnerusageedgenode"></a>`node` | [`CiDedicatedHostedRunnerUsage`](#cidedicatedhostedrunnerusage) | The item at the end of the edge. |
#### `CiGroupConnection`
 
The connection type for [`CiGroup`](#cigroup).
......@@ -21323,6 +21364,20 @@ CI/CD config variables.
| <a id="ciconfigvariablevalue"></a>`value` | [`String`](#string) | Value of the variable. |
| <a id="ciconfigvariablevalueoptions"></a>`valueOptions` | [`[String!]`](#string) | Value options for the variable. |
 
### `CiDedicatedHostedRunnerUsage`
Compute usage data for hosted runners on GitLab Dedicated.
#### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="cidedicatedhostedrunnerusagebillingmonth"></a>`billingMonth` | [`String!`](#string) | Month of the usage data. |
| <a id="cidedicatedhostedrunnerusagebillingmonthiso8601"></a>`billingMonthIso8601` | [`ISO8601Date!`](#iso8601date) | Timestamp of the billing month in ISO 8601 format. |
| <a id="cidedicatedhostedrunnerusagecomputeminutes"></a>`computeMinutes` | [`Int!`](#int) | Total compute minutes used across all namespaces. |
| <a id="cidedicatedhostedrunnerusagedurationseconds"></a>`durationSeconds` | [`Int!`](#int) | Total duration in seconds of runner usage. |
| <a id="cidedicatedhostedrunnerusagerootnamespace"></a>`rootNamespace` | [`Namespace`](#namespace) | Namespace associated with the usage data. Null for instance aggregate data. |
### `CiDurationStatistics`
 
Histogram of durations for a group of CI/CD jobs or pipelines.
......@@ -41252,6 +41307,15 @@ Values for sorting releases belonging to a group.
| <a id="groupreleasesortreleased_at_asc"></a>`RELEASED_AT_ASC` | Released at by ascending order. |
| <a id="groupreleasesortreleased_at_desc"></a>`RELEASED_AT_DESC` | Released at by descending order. |
 
### `GroupingEnum`
Values for grouping compute usage data.
| Value | Description |
| ----- | ----------- |
| <a id="groupingenuminstance_aggregate"></a>`INSTANCE_AGGREGATE` | Aggregate usage data across all namespaces in the instance. |
| <a id="groupingenumper_root_namespace"></a>`PER_ROOT_NAMESPACE` | Group data by individual root namespace. |
### `HealthStatus`
 
Health status of an issue or epic.
# frozen_string_literal: true
# rubocop:disable Gitlab/EeOnlyClass -- This is only used in GitLab dedicated that comes under ultimate tier only.
module EE
module Resolvers
module Ci
module Minutes
class DedicatedMonthlyUsageResolver < ::Resolvers::BaseResolver
include ::Gitlab::Graphql::Authorize::AuthorizeResource
include LooksAhead
type ::EE::Types::Ci::Minutes::DedicatedMonthlyUsageType.connection_type, null: true
argument :billing_month, GraphQL::Types::ISO8601Date, required: false,
description: 'First day of the month to retrieve data for.'
argument :year, GraphQL::Types::Int, required: false,
description: 'Year to retrieve data for.'
argument :grouping, EE::Types::Ci::Minutes::GroupingEnum, required: true,
description: 'Groups usage data by instance aggregate or root namespace.'
def resolve(**args)
grouping = args[:grouping]
billing_month = args[:billing_month]
year = args[:year]
case grouping
when 'INSTANCE_AGGREGATE'
resolve_instance_aggregate(billing_month, year)
when 'PER_ROOT_NAMESPACE'
resolve_per_root_namespace(billing_month, year)
end
end
private
def resolve_instance_aggregate(billing_month, year)
::Ci::Minutes::GitlabHostedRunnerMonthlyUsage
.instance_aggregate(billing_month, year).to_a
end
def resolve_per_root_namespace(billing_month, year)
::Ci::Minutes::GitlabHostedRunnerMonthlyUsage
.per_root_namespace(billing_month, year).to_a
end
end
end
end
end
end
# rubocop:enable Gitlab/EeOnlyClass
# frozen_string_literal: true
# rubocop:disable Gitlab/EeOnlyClass -- This is only used in GitLab dedicated that comes under ultimate tier only.
module EE
module Types
module Ci
module Minutes
class DedicatedMonthlyUsageType < ::Types::BaseObject
graphql_name 'CiDedicatedHostedRunnerUsage'
description 'Compute usage data for hosted runners on GitLab Dedicated.'
authorize :read_dedicated_hosted_runner_usage
field :billing_month, GraphQL::Types::String, null: false,
method: :billing_month_formatted,
description: 'Month of the usage data.'
field :billing_month_iso8601, GraphQL::Types::ISO8601Date, null: false, # rubocop:disable GraphQL/ExtractType -- we need it separate
description: 'Timestamp of the billing month in ISO 8601 format.'
field :compute_minutes, GraphQL::Types::Int, null: false,
description: 'Total compute minutes used across all namespaces.'
field :duration_seconds, GraphQL::Types::Int, null: false,
description: 'Total duration in seconds of runner usage.'
field :root_namespace, ::Types::NamespaceType, null: true,
description: 'Namespace associated with the usage data. Null for instance aggregate data.'
end
end
end
end
end
# rubocop:enable Gitlab/EeOnlyClass
# frozen_string_literal: true
# rubocop:disable Gitlab/EeOnlyClass -- This is only used in GitLab dedicated that comes under ultimate tier only.
module EE
module Types
module Ci
module Minutes
class GroupingEnum < ::Types::BaseEnum
graphql_name 'GroupingEnum'
description 'Values for grouping compute usage data.'
value 'INSTANCE_AGGREGATE', 'Aggregate usage data across all namespaces in the instance.'
value 'PER_ROOT_NAMESPACE', 'Group data by individual root namespace.'
end
end
end
end
end
# rubocop:enable Gitlab/EeOnlyClass
......@@ -33,6 +33,23 @@ module QueryType
required: false,
description: 'Date for which to retrieve the usage data, should be the first day of a month.'
end
field :ci_dedicated_hosted_runner_usage, Types::Ci::Minutes::DedicatedMonthlyUsageType.connection_type,
null: true,
resolver: EE::Resolvers::Ci::Minutes::DedicatedMonthlyUsageResolver,
description: 'Compute usage data for runners across namespaces on GitLab Dedicated. ' \
'Defaults to the current year if no year or billing month is specified. ' \
'Ultimate only.' do
argument :billing_month, ::Types::DateType,
required: false,
description: 'First day of the month to retrieve data for.'
argument :year, GraphQL::Types::Int,
required: false,
description: 'Year to retrieve data for.'
argument :grouping,
type: EE::Types::Ci::Minutes::GroupingEnum,
required: false,
description: 'Groups usage data by instance aggregate or root namespace.'
end
field :current_license, ::Types::Admin::CloudLicenses::CurrentLicenseType,
null: true,
resolver: ::Resolvers::Admin::CloudLicenses::CurrentLicenseResolver,
......
......@@ -26,8 +26,45 @@ class GitlabHostedRunnerMonthlyUsage < Ci::ApplicationRecord
}
validate :validate_billing_month_format
scope :instance_aggregate, ->(billing_month, year) do
select("TO_CHAR(billing_month, 'FMMonth YYYY') AS billing_month_formatted",
'billing_month AS billing_month',
'TO_CHAR(DATE_TRUNC(\'month\', billing_month), \'YYYY-MM-DD\') AS billing_month_iso8601',
'SUM(compute_minutes_used) AS compute_minutes',
'SUM(runner_duration_seconds) AS duration_seconds',
'NULL as root_namespace_id')
.where(billing_month: billing_month_range(billing_month, year))
.group(:billing_month)
.order(billing_month: :desc)
end
scope :per_root_namespace, ->(billing_month, year) do
where(billing_month: billing_month_range(billing_month, year))
.group(:billing_month, :root_namespace_id)
.select("TO_CHAR(billing_month, 'FMMonth YYYY') AS billing_month_formatted",
'billing_month AS billing_month',
'TO_CHAR(DATE_TRUNC(\'month\', billing_month), \'YYYY-MM-DD\') AS billing_month_iso8601',
'root_namespace_id',
'SUM(compute_minutes_used) AS compute_minutes',
'SUM(runner_duration_seconds) AS duration_seconds')
.order(billing_month: :desc, root_namespace_id: :asc)
end
private
def self.billing_month_range(billing_month, year)
if billing_month.present?
start_date = billing_month
end_date = start_date.end_of_month
else
year ||= Time.current.year
start_date = Date.new(year, 1, 1)
end_date = Date.new(year, 12, 31)
end
start_date..end_date
end
def validate_billing_month_format
return if billing_month.blank?
......
# frozen_string_literal: true
module Ci
module Minutes
class GitlabHostedRunnerMonthlyUsagePolicy < BasePolicy
desc "User is an admin on GitLab Dedicated"
condition :gitlab_dedicated do
Gitlab::CurrentSettings.gitlab_dedicated_instance?
end
rule { gitlab_dedicated & admin }.policy do
enable :read_dedicated_hosted_runner_usage
end
end
end
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe EE::Resolvers::Ci::Minutes::DedicatedMonthlyUsageResolver, feature_category: :hosted_runners do
include GraphqlHelpers
let_it_be(:user) { create(:user) }
let_it_be(:admin) { create(:admin) }
let_it_be(:namespace1) { create(:namespace) }
let_it_be(:namespace2) { create(:namespace) }
let_it_be(:billing_month) { Date.new(2025, 1, 1) }
let_it_be(:billing_month2) { Date.new(2025, 2, 1) }
let_it_be(:usage1) do
create(:ci_hosted_runner_monthly_usage, billing_month: billing_month, compute_minutes_used: 100,
runner_duration_seconds: 6000, root_namespace: namespace1)
end
let_it_be(:usage2) do
create(:ci_hosted_runner_monthly_usage, billing_month: billing_month, compute_minutes_used: 200,
runner_duration_seconds: 12000, root_namespace: namespace2)
end
let_it_be(:usage3) do
create(:ci_hosted_runner_monthly_usage, billing_month: billing_month2, compute_minutes_used: 100,
runner_duration_seconds: 6000, root_namespace: namespace1)
end
let_it_be(:usage4) do
create(:ci_hosted_runner_monthly_usage, billing_month: billing_month2, compute_minutes_used: 200,
runner_duration_seconds: 12000, root_namespace: namespace2)
end
let(:year) { nil }
let(:args) { { billing_month: billing_month_arg, year: year, grouping: grouping } }
before do
stub_application_setting(gitlab_dedicated_instance: true)
end
describe '#resolve' do
context 'when grouping is INSTANCE_AGGREGATE', :enable_admin_mode do
let(:grouping) { 'INSTANCE_AGGREGATE' }
let(:billing_month_arg) { billing_month }
it 'returns the correct instance aggregate data' do
result = resolve_usage(admin)
expect(result.count).to eq(1)
expect(result.first.compute_minutes).to eq(usage1.compute_minutes_used + usage2.compute_minutes_used)
expect(result.first.duration_seconds).to eq(usage1.runner_duration_seconds + usage2.runner_duration_seconds)
expect(result.first.root_namespace).to be_nil
end
context 'when year is passed', :enable_admin_mode do
let(:billing_month_arg) { nil }
let(:year) { 2025 }
it 'returns the correct instance aggregate data for all months in the year' do
result = resolve_usage(admin)
expect(result.count).to eq(2)
expect(result.map(&:billing_month_iso8601)).to match_array([
billing_month.iso8601,
billing_month2.iso8601
])
expect(result.sum(&:compute_minutes)).to eq(
usage1.compute_minutes_used + usage2.compute_minutes_used +
usage3.compute_minutes_used + usage4.compute_minutes_used
)
expect(result.sum(&:duration_seconds)).to eq(
usage1.runner_duration_seconds + usage2.runner_duration_seconds +
usage3.runner_duration_seconds + usage4.runner_duration_seconds
)
expect(result.first.root_namespace).to be_nil
end
end
end
context 'when grouping is PER_ROOT_NAMESPACE', :enable_admin_mode do
let(:grouping) { 'PER_ROOT_NAMESPACE' }
let(:billing_month_arg) { billing_month }
it 'returns the correct data per root namespace' do
result = resolve_usage(admin)
expect(result.count).to eq(2)
expect(result.map(&:root_namespace_id)).to match_array([namespace1.id, namespace2.id])
namespace1_usage = result.find { |usage| usage.root_namespace_id == namespace1.id }
namespace2_usage = result.find { |usage| usage.root_namespace_id == namespace2.id }
expect(namespace1_usage.compute_minutes).to eq(usage1.compute_minutes_used)
expect(namespace2_usage.compute_minutes).to eq(usage2.compute_minutes_used)
end
context 'when year is passed', :enable_admin_mode do
let(:billing_month_arg) { nil }
let(:year) { 2025 }
it 'returns the correct data per root namespace for all months in the year' do
result = resolve_usage(admin)
expect(result.count).to eq(4)
expect(result.map(&:billing_month_iso8601)).to match_array([
billing_month2.iso8601,
billing_month2.iso8601,
billing_month.iso8601,
billing_month.iso8601
])
expect(result.sum(&:compute_minutes)).to eq(
usage1.compute_minutes_used +
usage2.compute_minutes_used +
usage3.compute_minutes_used +
usage4.compute_minutes_used
)
expect(result.sum(&:duration_seconds)).to eq(
usage1.runner_duration_seconds +
usage2.runner_duration_seconds +
usage3.runner_duration_seconds +
usage4.runner_duration_seconds
)
end
end
end
context 'when user is not on GitLab Dedicated', :enable_admin_mode do
let(:grouping) { 'INSTANCE_AGGREGATE' }
let(:billing_month_arg) { billing_month }
before do
stub_application_setting(gitlab_dedicated_instance: false)
end
it 'returns no data' do
result = resolve_usage(admin)
expect(result).to be_empty
end
end
context 'when user is not an admin' do
let(:grouping) { 'INSTANCE_AGGREGATE' }
let(:billing_month_arg) { billing_month }
it 'returns no data' do
result = resolve_usage(user)
expect(result).to be_empty
end
end
def resolve_usage(current_user)
resolve(described_class, obj: nil, args: args, ctx: { current_user: current_user }).to_a
end
end
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe GitlabSchema.types['CiDedicatedHostedRunnerUsage'], feature_category: :hosted_runners do
include GraphqlHelpers
subject { described_class }
let_it_be(:fields) { %i[billing_month billing_month_iso8601 compute_minutes duration_seconds root_namespace] }
it { is_expected.to have_graphql_fields(fields) }
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe GitlabSchema.types['GroupingEnum'], feature_category: :continuous_integration do
specify { expect(described_class.graphql_name).to eq('GroupingEnum') }
it 'exposes all the grouping strategy values' do
expect(described_class.values.keys).to include(
*%w[INSTANCE_AGGREGATE PER_ROOT_NAMESPACE]
)
end
end
......@@ -15,6 +15,7 @@
:ci_catalog_resources,
:ci_catalog_resource,
:ci_minutes_usage,
:ci_dedicated_hosted_runner_usage,
:ci_queueing_history,
:current_license,
:devops_adoption_enabled_namespaces,
......
......@@ -69,4 +69,59 @@
end
end
end
describe '.instance_aggregate' do
let(:billing_month) { Date.new(2025, 1, 1) }
subject(:instance_aggregate) { described_class.instance_aggregate(billing_month, nil).to_a }
it 'returns the correct aggregate data' do
create(:ci_hosted_runner_monthly_usage,
billing_month: billing_month,
compute_minutes_used: 100,
runner_duration_seconds: 6000)
create(:ci_hosted_runner_monthly_usage,
billing_month: billing_month,
compute_minutes_used: 200,
runner_duration_seconds: 12000)
expect(instance_aggregate.count).to eq(1)
expect(instance_aggregate.first.compute_minutes).to eq(300)
expect(instance_aggregate.first.duration_seconds).to eq(18000)
expect(instance_aggregate.first.root_namespace_id).to be_nil
end
end
describe '.per_root_namespace' do
let(:billing_month) { Date.new(2025, 5, 1) }
let(:namespace1) { create(:namespace) }
let(:namespace2) { create(:namespace) }
subject(:per_root_namespace) { described_class.per_root_namespace(billing_month, nil).to_a }
it 'returns the correct data per root namespace' do
create(:ci_hosted_runner_monthly_usage,
billing_month: billing_month,
compute_minutes_used: 100,
runner_duration_seconds: 6000,
root_namespace: namespace1)
create(:ci_hosted_runner_monthly_usage,
billing_month: billing_month,
compute_minutes_used: 200,
runner_duration_seconds: 12000,
root_namespace: namespace2)
expect(per_root_namespace.count).to eq(2)
expect(per_root_namespace.map(&:root_namespace_id))
.to match_array([namespace1.id, namespace2.id])
expect(per_root_namespace.find do |usage|
usage.root_namespace_id == namespace1.id
end.compute_minutes).to eq(100)
expect(per_root_namespace.find do |usage|
usage.root_namespace_id == namespace2.id
end.compute_minutes).to eq(200)
end
end
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Ci::Minutes::GitlabHostedRunnerMonthlyUsagePolicy, feature_category: :hosted_runners do
let_it_be_with_reload(:current_user) { create(:admin) }
let(:gitlab_hosted_runner_monthly_usage) { create(:ci_hosted_runner_monthly_usage) }
subject(:policy) { described_class.new(current_user, gitlab_hosted_runner_monthly_usage) }
context 'when GitLab instance is not dedicated' do
before do
stub_application_setting(gitlab_dedicated_instance: false)
end
it { is_expected.not_to be_allowed(:read_dedicated_hosted_runner_usage) }
end
context 'when GitLab instance is dedicated' do
before do
stub_application_setting(gitlab_dedicated_instance: true)
end
context 'when user is an admin', :enable_admin_mode do
it { is_expected.to be_allowed(:read_dedicated_hosted_runner_usage) }
end
context 'when user is not an admin' do
let(:current_user) { create(:user) }
it { is_expected.not_to be_allowed(:read_dedicated_hosted_runner_usage) }
end
end
end
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment