Research: Investigate the validity of the instrumentation for secrets installation counts...

Summary

Currently, we have a Sisense widget that captures the total unique count of installations with Secrets Management Vault. As we move towards adding more instrumentation with secret providers and JWT token usage, measuring its success and adoption will require that the counts are accurate.

The concern is how accurate this data as a whole as the number of namespaces overall seems to be low (only showing in the 100s). The conversation in Slack can be seen here.

Research

The effort here is to investigate and validation that the instrumentation that is being populated in that Sisense widget is correct. Since we are releasing a new version of the JWT, maybe its worth examining the way we collect the data from our users to track its adoption.

Findings

The instrumentation appears to be valid for Secrets. The tracking happens when a Build job is created. However, it's currently implemented with Service Ping which has limitations in the SaaS space. So the data we see in the Sisense widget pertains only to Self-Managed instances. To provide data coverage for SaaS namespaces, it's recommended that we implement additional instrumentation with Snowplow.

Separate issues have been created to implement both Service Ping and Snowplow data tracking for Secrets and JWT Token usage.

See this thread for details on the research.