RFC: What is a trial?
Request for comments
Need
You've seen the word. You've may have felt the feelings. You may have even said "The number of trials is too damn high!" with the .gif
of Jimmy McMillan in your brain.
What the heck is a "trial" though?
We need to introduce some shared language and understanding around the words we use to mean everything from "trying a new idea out" to "rolling out a change that we'll iterate on in the future".
Approach
Let's discuss. I'll put some thoughts here, but this is an RFC so I'm more interested in the discussion.
Here's a collection of frustrations I've seen expressed over the years. Feel free to add some.
Science, but business.
One frequent issue with "trials" is the spectrum of what data-driven decision making means to folks. Some of us want all decisions to have scientific merit: isolation of variables, a control group and rigorous data standards.
Others of us have a more lean software approach where we want to move fast, get things out there and iterate when needed.
There's a spectrum there that probably needs to be applied different for different levels of risk / challenge of change.
How many trials can happen at once?
We've seen situations where people are involved in multiple trials, or where two trials affect one another directly and the sponsors don't seem to be talking to one another.
Who is driving the priority? Can I just make a random trial because I want to?
Sometimes the spirit of the age drives someone to work on something that doesn't have a direct tie to a department or company objective.
Why do I have to wait for a rollout to try this?
We've seen situations where results are delayed because of the rollout process on the Ops side.
I'd like to explore this as a potential taxonomy. It doesn't solve everything above, but I think it might help.
Phase | Description | Tooling Needs | Data needs | Training needs | Difficulty of a roll-back | Leadership awareness |
---|---|---|---|---|---|---|
Experiment | I have an idea I want to try. I will try it by myself, or get a small group of people to try it with me. | Self-built or non-existent and relies on manual process | Self-reported, experiential | Zero to a brief conversation with participants | Seamless - you just stop doing the thing you were doing | Manager informed |
Trial | Experiment went well, I want to try it at a larger scale based on the experience we had. | Self-built, non-existent and relies on manual process. | Self-reported, experiential is acceptable, but starting to dashboard and collect more empirical data | Some documentation in an issue and agreement from participants that they'll follow it | Nearly seamless - you have to coordinate a larger group, but by and large relatively easy because you were relying on manual processes | Manager sponsored |
Pilot | Trial went well, and we want to pilot the full experience with proper tooling support. | MVC changes that affect a larger group built, scheduled and rolled out by Ops in coordination with the group running the pilot | Clear success criteria and measures of success, collaboration with Readiness to ensure data quality | Manager-involved in making sure all participants are read in and understand the objectives and procedures | Challenging | Manager-led, Director informed |
Rollout | Pilot went well, and we want this to be fully adopted | Fully developed, tested solution | PIs are locked in and well understood | Training module developed and there's a team requirement for completing it | Very, very challenging as this is now likely affecting data across the organization. Likely impossible and will require a roll-forward with a new iteration. | Director sponsored |
Benefit
- Better understanding of who needs to be involved where
- Shared language where we're all referring to the same thing