Skip to content

Clarify details about experiments, beta features, and feature flags

Summary

This issue is a summary of challenges raised by a thread in the #product slack channel (screenshots documented in the comments below). There are multiple challenges related to the naming of releases and the documentation outcomes from those naming conventions. I've mapped them out as separate challenges with potential courses of action below.

Challenge 1: Releases for a specific audience

We currently have features that are available only to a subset of users (team members only or specific customers only).

Question: Do we want to do this? (If yes, the next questions follow up on that decision.)

Challenge 2: Documentation for a specific audience

If we're going to release features to specific audiences, we should formalize these terms or update the docs to have more visible audience information. We have been using terms like closed access, open alpha, limited access, or internal experiment. However, these terms are not documented anywhere. These alternate terms are see in issues, epics, documentation, and field guides, which is causing confusion for both internal and external stakeholders. For example: Implementation teams don’t have an accurate handbook reference that shows the clear release / quality standards for these new terms.

Question: Do we want to document some new "audience" terms on the Experiment and Beta support page? Or should we make audience information more visible in the docs (special notes, other)? Or should we not document features that aren't available to everyone?

Challenge 3: Feature flags and feature statuses

We have guidance that turning on a feature flag is too much of a burden for experimental features. ("Needing to flip a feature flag is too much friction") However, generally the process is:

  • Experiment: Feature flag disabled by default
  • Beta: Feature flag enabled by default
  • GA: Feature flag removed

With the recent updates to our Minimum Valuable Change guidance, this guidance may no longer be practical or desirable. It may be that we need a fourth term that indicates “internal only” as part of the definition. A change of this nature would need broad buy-in from leadership.

Question: Should we enforce this flow? Or formalize a different flow? Should we change the guidance about experiments and flags?

Challenge 4: Should every feature be documented?

Right now, every feature is documented, even those behind flags. (ie. If / when we do internal-only releases, we are updating our public-facing documentation as well.) The reasoning is that the docs and code go together in the first MR, and that way the docs weren't forgotten.

Current tagging of “internal only” on experiments isn’t immediately obvious (ex: duo workflow) and customers / sales teams may waste time trying to set up and use the feature.

Question: Should we change this guidance? If we do, what determines when a feature should be documented?

Challenge 5: Customer and internal guidance about experiments and beta

There is information on the experiments and beta page that should be in the handbook or under Contribute. (Everything here and below.)

Question: Should we move the internal content? We can link the pages. Or can we make an edit so the page is clearly marked to delineate between internal and customer-facing? We might want to review the whole page to ensure our customer guidance and internal guidance don't have conflicts.

Immediate Next Steps

  • We need a Product Team DRI to drive this change - @natalie.pinto will talk to PLT about finding a PM with bandwidth to drive this.

Potential Solution Paths

Note: the potential solution paths listed here are not a complete list. This is a record of some ideas that have been thrown out, but the eventual DRI for this should evaluate these ideas and solicit others before finalizing a plan.

  • Audit of terms used across docs, issues, epics, public releases, etc to understand the scope of terms that may need to be changed, and where.
  • Decide if we could benefit from a fourth term that indicates “internal only” as part of the definition.
  • Evaluate if an audience indicator of some kind in the documentation would be a better solution than a new internal-only focused naming convention.
  • Evaluate how / if the Minimum Valuable Change MR can be leveraged to advocate for an internal-only release naming convention.
  • Pending this decision, change terminology as needed throughout docs, issues, epics, public releases, etc. Align the AI field guide to match our internal definition schema as well.
  • Validate the outcome of any decisions made about internal-only releases with e-group (Sid especially).
Edited by Natalie Pinto