Streamline AI Logging

Description

We need to improve and standardize our AI logging practices to better support both internal debugging and customer needs, especially for self-hosted instances.

Objectives

1. Consolidate and document events in llm.log => Consolidate and Document llm.log (#477469 - closed) • Gosia Ksionek • 17.5
2. Ensure consistent logging format across AI features
3. Provide guidance on what should and shouldn't be logged

Tasks

1. Audit current logging practices across AI features => https://internal.gitlab.com/handbook/product/ai-strategy/duo-logging/#current-state-of-ai-logging
2. Define standard logging format and required fields
3. Document all events being saved in llm.log
4. Identify and add missing fields/events
5. Create guidelines for teams on what to log (and what not to log)
6. Implement consistent logging in reusable AI components
7. Update documentation on llm.log usage and interpretation

Considerations

• Potential need to split logs (debug vs. customer-facing) in the future
• Data privacy and security (e.g., not logging prompts, user IDs)
• Self-hosted customer requirements
• Performance impact of expanded logging

Current state of LLM logging

Event message	description	class	rails	sidekiq	arguments	Part of the system	Expanded logging?
Returning from Service due to validation	user not permitted to perform action	Llm::BaseService	yes	no	none	Abstraction layer	no
Enqueuing CompletionWorker	scheduling completion worker in sidekiq	Llm::BaseService	yes	no	user_id: message.user.id, resource_id: message.resource&.id,resource_class: message.resource&.class&.name,request_id: message.request_id,action_name: message.ai_action,options: job_options	Abstraction layer	yes
Returning from Service due to missing resource id. Associated group: #{@group}	If there is no resource for slash command	Llm::ChatService	yes	no	none	Chat	no
Got Completion Service from factory	Completion Service found	Llm::Internal::CompletionService	no	yes	class_name: completion.class.name	Abstraction layer	yes
Performing CompletionService	performing completion	Llm::Internal::CompletionService	no	yes	user_id: prompt_message.user.to_gid,resource_id: prompt_message.resource&.to_gid,action_name: prompt_message.ai_action,request_id: prompt_message.request_id,client_subscription_id: prompt_message.client_subscription_id	Abstraction layer	yes
Answer	Get answer from response	Gitlab::Llm::Chain::Answer	no	yes	content: content	chat	yes
Final answer	Get final answer from response	Gitlab::Llm::Chain::Answer	no	yes	content: content	chat	yes
Default final answer	"Default final answer: I'm sorry, I couldn't respond in time. Please try a more specific request or enter /clear to start a new chat."	Gitlab::Llm::Chain::Answer	no	yes	error_code: "A6000"	chat	yes
Error message	when answering with an error	Gitlab::Llm::Chain::Answer	no	yes	error: content, error_code: error_code	chat	no
Received response from AI Gateway	when response from AIGW is returned	Gitlab::Llm::AiGateway::Client	no	yes	response: response.parsed_response	Abstraction layer	yes
Received error from AI gateway	when error is returned from AIGW for streaming command	Gitlab::Llm::AiGateway::Client	no	yes	response: response_body	Abstraction layer	no
Performing request to AI Gateway	before performing request to the AI GW	Gitlab::Llm::AiGateway::Client	no	yes	body: body, timeout: timeout,stream: stream"	Abstraction layer	yes
Creating user access token	creating short-lived token in AIGW	Gitlab::Llm::AiGateway::CodeSuggestionsClient	yes	no		code suggestions	no
Received response from Anthropic	Received response in complete method	Gitlab::Llm::Anthropic::Client	no	yes	response: response_completion	Abstraction layer	yes
Received response from Anthropic	Received response in stream complete method	Gitlab::Llm::Anthropic::Client	no	yes	response: response_completion	Abstraction layer	yes
Received response from Anthropic	Received response in stream messages method	Gitlab::Llm::Anthropic::Client	no	yes	response: response_completion	Abstraction layer	yes
Received response from Anthropic	Received response in messages method	Gitlab::Llm::Anthropic::Client	no	yes	response: response_completion	Abstraction layer	yes
Performing request to Anthropic	performing completion request	Gitlab::Llm::Anthropic::Client	no	yes	options: options	Abstraction layer	no
Performing request to Anthropic	performing messages request	Gitlab::Llm::Anthropic::Client	no	yes	options: options	Abstraction layer	no
Searching docs from AI Gateway	performing search docs request	Gitlab::Llm::AiGateway::DocsClient	no	yes	options: options	chat	no
Searched docs from AI Gateway	response from AIGW with docs	Gitlab::Llm::AiGateway::DocsClient	no	yes	response: response	chat	yes
Error	logged when json is not parsable	Gitlab::Llm::Anthropic::Completions::CategorizeQuestions	no	yes	class: self.class.to_s, error: error_message	chat	no
Error	logged when response is not containing one of the defined categories	Gitlab::Llm::Anthropic::Completions::CategorizeQuestions	no	yes	class: self.class.to_s, error: error_message	chat	no
Picked tool	information about tool picked by chat	Gitlab::Llm::Chain::Agents::ZeroShot::Executor	no	yes	tool: tool_class.to_s	chat	no
Made request to AI Client	making request for chat	Gitlab::Llm::Chain::Requests::AiGateway	no	yes	class: self.class.to_s, prompt: prompt[:prompt], response: response	chat	yes
Streaming error	Error returned when streaming	Gitlab::Llm::Chain::Requests::Anthropic	no	yes	error: data&.dig("error")	old implementation of summarize comments	no
Got Final Result	got result for documentation question	Gitlab::Llm::Chain::Tools::EmbeddingsCompletion	no	yes	prompt: final_prompt[:prompt], response: final_prompt_result	chat	yes
Streaming error	when error is returned from AIGW for streaming command in docs question	Gitlab::Llm::Chain::Tools::EmbeddingsCompletion	no	yes	error: error.message	chat	no
Answer	"when tool was already picked up (content: You already have the answer from #{self.class::NAME} tool, read carefully.)"	Gitlab::Llm::Chain::Tools::Tool	no	yes	class: self.class.to_s, content: content	chat	yes
Tool cycling detected	When tool is picked up again	Gitlab::Llm::Chain::Tools::Tool	no	yes		chat	no
Calling TanukiBot	performing documentation request	Gitlab::Llm::Chain::Tools::GitlabDocumentation::Executor	no	yes	class: self.class.to_s	chat	no
Error finding #{resource_name}	when resource (issue/epic/mr) is not found	Gitlab::Llm::Chain::Tools::Identifier	no	yes	content: json	chat	no
Answer	response from identifier	Gitlab::Llm::Chain::Tools::Identifier	no	yes	class: self.class.to_s, content: content	chat	yes
Error	when json is malformed (Observation: JSON has an invalid format. Please retry)	Gitlab::Llm::Chain::Tools::Identifier	no	yes	class: self.class.to_s, error: error_message	chat	no
Answer	"already identified resource (You already have identified the #{resource_name} #{resource.to_global_id}, read carefully.)"	Gitlab::Llm::Chain::Tools::Identifier	no	yes	class: self.class.to_s, content: content	chat	yes
Supported Issuable Typees Ability Allowed	logging the ability (policy.can?) for the issue/epic	Gitlab::Llm::Chain::Tools::SummarizeComments::Executor	no	yes	content: ability	chat	yes
Supported Issuable Typees Ability Allowed	logging the ability (policy.can?) for the issue/epic	Gitlab::Llm::Chain::Tools::SummarizeComments::ExecutorOld	no	yes	content: ability	summarize comment feature	yes
Answer	Answer for summarize comments feature	Gitlab::Llm::Chain::Tools::SummarizeComments::ExecutorOld	no	yes	class: self.class.to_s, content: content	summarize comment feature	yes
Prompt	chat-related request	Gitlab::Llm::Chain::Concerns::AiDependent	no	yes	class: self.class.to_s, prompt: prompt_text	chat	yes
"""Too many requests, will retry in #{delay} seconds"""	When entered in exponential backoff loop	Gitlab::Llm::Chain::Concerns::ExponentialBackoff	no	yes		Abstraction layer	no
Resource not found	Resource not found/not authorized	Gitlab::Llm::Utils::Authorizer	no	yes	error_code: "M3003"	Abstraction layer	yes
No access to Duo Chat	No access to duo chat	Gitlab::Llm::Utils::Authorizer	no	yes	error_code: "M3004"	Abstraction layer	yes
AI is disabled	AI is not enabled for container	Gitlab::Llm::Utils::Authorizer	no	yes	error_code: "M3002"	Abstraction layer	yes
Performing request to Vertex	performing request	Gitlab::Llm::VertexAi::Client	no	yes	config: config	Abstraction layer	no
Received response from Vertex	response from aigw - vertex	Gitlab::Llm::VertexAi::Client	no	yes	response: response.to_json	Abstraction layer	yes
Empty response from Vertex	empty response from aigw - vertex	Gitlab::Llm::VertexAi::Client	no	yes		Abstraction layer	no