Update CycloneDX report ingestion to track feature usage

Proposal

We currently lack metrics about our CycloneDX based features, and as a result have a hard time prioritizing feature requests. We're also unable to proactively detect things in the analyzer like performance issues until they're reported to us by users. Here are some metrics that can be helpful.

  • Tool name and version that produced an SBOM
    • Answers questions like "What's the impact of deprecating this analyzer?"
    • Gives us insight to the competitive landscape. If customers are using a different analyzer, we should ask why that is, and if there's something we can improve.
  • Detection metrics
    • Duration, CPU, and memory consumption
    • Job status, warnings, and errors
      • Can help us catch bugfunctional regressions.
      • Helpful in correlating scan failures with a cause. For example, we could correlate a pip-compile error increase with the rollout of a new analyzer version.
  • Analyzer features
    • Flag usage and env variable usage.
    • Helps us answer questions like "How many people use this flag?" and even allows us to gauge how --experimental flags are doing.
  • Track usage of CycloneDX report features like vulnerabilities and licenses.
    • Helps us determine how many users would take advantage of such features. Take #504040 (closed) and #435436 for example.

Events and Metrics

Definition of events and metrics | Sample metrics | Sample events

Details of events to be tracked:

Event Description Event Name Additional Properties Feature

Details of metrics to be tracked:

Metric Description Event / DB column to base the Metric on Total or Unique Count of a Property Time Frame Feature

Expand to view examples and guidelines for filling the table

Events:

  • Description: Include what the event is supposed to track, where and when.
  • Name: Primary identifier of the event, format: <action>_<target_of_action>_<where/when>
    • Example event name: click_save_button_in_issue_description_within_15s_of_page_load (action = click ; target = save button; where = in issue description ; when = within 15s
  • **Additional properties: Besides user/project/namespace, what other details should be tracked, if any? ex) status, type, object id, etc.
  • Feature: What feature is being instrumented? Please use the feature title that is used in features.yml if thats already available.

Metrics:

  • Description: What quantitative measurements derived from either event data or database columns would you like to track? eg: Weekly count of unique users who update an issue
  • Event/DB column: What event or database column should the metric count or be based on.
  • Total or unique count: Should the metric count all occurrences or only unique counts, e.g. of user_id to get a count of unique users triggering an event.
  • Time Frame: What time frames should be tracked. Default and recommended is 7d and 28d.

Next steps

  • Assign an engineering counterpart from your group to add instrumentation to the code
  • Explore instrumented data with the help of our data discovery guide. You can also reach out to product data insights team for help with generating Tableau reports/dashboards.
  • Your feedback is valuable to us. Please leave us feedback in the comment section of this issue and tag @tjayaramaraju.

Important links

⭐ Quick start guide to internal events

❓ Analytics Instrumentation slack channel for questions

✍️ Try our internal events generator. Creating event and metric definition files has never been easier.

Edited by Oscar Tovar