UX Experience for Self-Hosted Auditing / Logging

The Problem

As Custom Model moves towards developing self-hosted model support and customization approaches, we need to allow customers visibility into their own LLM flows for debugging, auditing, validation, and potentially accumulating data sets for supervised fine tuning. As we enable customer facing logging, there must be a clear way for customers to visualize and understand that data.

Use Cases

• auditing
• debugging
• validation
• datasets for supervised fine tuning: customers will likely require between 6-8k examples of 'good' prompt and responses in order to optimally shift the weights of pre-trained models for their own use cases. Our hypothesis is that customers likely do not have these datasets at hand, but access to their own logs as they begin to implement LLM operators in their own environments could help them build those datasets for later curation and use in supervised fine tuning (SFT)
• anything that could materially change the privacy, security, or enable/disable features should have audit log events.

The Proposal

• Customers have a dashboard view they can use to see and pivot on all available fields, to potentially include:
  • token usage (input / output)
  • user prompt / LLM response
  • latency (timestamps for user prompt / LLM response)
  • user group
  • user project
  • user project supergroups
  • code owner
  • data security classification level of the user’s current file, repo, group
  • manual selection of input routing (see end user controlled model routing)
  • context-based parameters such as PII detection or zero shot determination (see context based model routing)
• Customers will have to opt into this capture of data, and can opt into particular facets
• Customers can configure their own retention period with options: 30 days, 90 days, 120 days, and 'no age off'

What This Work Will Inform

The information that we capture and make available to end users will inform future work, to include model routing and validation.