Streaming audit events to Google Cloud Logging
## Problem Statement
Enable Group admins to configure google cloud logging and stream audit events to it.
## Proposed Solution
Users can configure an IAM service account under their project in google cloud and provide the details required for generating the access token: private_key, client_email, and project_id. using these we can generate an access token and stream audit events to the google cloud logging service. Users need to make sure the IAM service account has logging writes.
## Proposed Implementation Plan
gitlab~2492649
1. Create a new table to store Google Cloud configurations, which should have project_id, client_email, and private_key. Note: store the private key securely in encrypted form. In the migration below, I have commented out what I used for the POC and what's recommended for the actual MR.
```ruby
create_table :audit_events_google_cloud_logging_configurations do |t|
t.bigint :group_id, null: false
t.text :google_project_id, null: false # t.text :project_id, null: false
t.text :client_email, null: false
t.text :log_name, default: "audit_events"
t.binary :encrypted_private_key, null: false #t.text :private_key, null: false
t.timestamps null: false
```
2. Create `AuditEvents::GoogleCloudLoggingConfiguration`
```ruby
module AuditEvents
class GoogleCloudLoggingConfiguration < ApplicationRecord
self.table_name = 'audit_events_google_cloud_logging_configurations'
belongs_to :group
end
end
```
3. Set up the association with the Group model.
```ruby
has_many :google_cloud_logging_configurations, class_name: "AuditEvents::GoogleCloudLoggingConfiguration", inverse_of: :group
```
4. Write GraphQL APIs to insert, update and delete data from this model. (This can be a separate issue) for gitlab~2492649 with weight estimate of 5.
4. Modify AuditEvents::AuditEventStreamingWorker to stream audit events to the Google Cloud Logging service if Google Cloud Logging configurations are present.
```ruby
group.google_cloud_logging_configurations.each do |google_cloud_config|
send_to_google_cloud_logging(google_cloud_config, audit_event, audit_operation)
end
```
5. To stream we will need to generate an access token
```ruby
def generate_access_token(client_email, private_key)
credentials = Google::Auth::ServiceAccountCredentials.make_creds(
json_key_io: StringIO.new({ client_email: client_email, private_key: private_key }.to_json),
scope: 'https://www.googleapis.com/auth/logging.write'
)
credentials.fetch_access_token!
end
```
6. The method below will POST audit events as JSON to the Google Cloud Logging service. This was written for the POC, and you can use it as a starting point for writing more refined code.
```ruby
# Dont use as it is, this was just for POC.
def send_to_google_cloud_logging(gc_config, audit_event, audit_operation)
puts "Sending to google cloud"
project_id = gc_config.project_id
log_name = "audit_events"
access_token = generate_access_token(gc_config.client_email, gc_config.private_key)["access_token"]
log_entry = {
'logName' => "projects/#{project_id}/logs/#{log_name}",
'resource' => {
'type' => 'global'
},
'severity' => 'INFO',
'jsonPayload' => JSON::parse(request_body(audit_event, audit_operation))
}
uri = URI("https://logging.googleapis.com/v2/entries:write")
request = Net::HTTP::Post.new(uri)
request['Authorization'] = "Bearer #{access_token}"
request['Content-Type'] = 'application/json'
request.body = { 'entries' => [log_entry] }.to_json
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
http.request(request)
end
puts response.code
puts response.body
if response.code.to_i >= 400
"Error sending log entry to Google Cloud Logging: #{response.body}"
end
end
```
According to the above implementation, we should divide this into three separate issues:
1. https://gitlab.com/gitlab-org/gitlab/-/issues/409421+s
2. https://gitlab.com/gitlab-org/gitlab/-/issues/409422+s
3. https://gitlab.com/gitlab-org/gitlab/-/issues/409423+
4. FE
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
epic