The Experiment Card provides a detailed guide on how to design, build, and measure experiments to validate business hypotheses or assumptions.
It explains the steps involved in picking potential tests, defining a key performance indicator (KPI) or "One Metric That Matters" (OMTM), sketching the experiment, defining and finding the target audience, and listing and ranking the best channels to find the audience.
The canvas also provides steps on how to translate the plan into actions and how to analyze and reflect on the results after running the experiment. It emphasizes the importance of documenting and reflecting on findings and updating your progress board.
“There’s many roads leading to Rome”, I’m sure you’re familiar with this quote. Similarly with all your business hypothesis or assumptions, there’s multiple ways to test and validate.
But since many entrepreneurs are not familiar with all possible ways, that’s why we launched “Validation Cards” in Bites. Bites is a set of tips, tricks and inspirations to help you make the right possible assumption.
<aside> 👉 Access 50+ Bites here: www.febe.io/bites/home Customer who bought the StartHack Framework are automatically granted a full one year FREE subscription.
Simply use the discount code sent to your email when you signed up for StartHack in your checkout or reach out to our support [email protected]
</aside>
We want you to grab 3 test you think are the best to validate your assumption. We, then, want to rank the best one. We can then execute one (and have the rest as back-up plan) or test multiple ones and compare results. The best way to rank experiments is to use the SIR score:
Score them from 1-5 and then use this formula: S*R/I
Now, that you have your top test ranked, it’s time to define what success looks like and answer. In short: how do we know we successfully validated our idea?
We do that by defining OMTM (One Metric That Matters). It’s just another definition to a KPI, but we wanted to call it OMTM to emphasize on making it ONE.
How to find the right KPI for your experiment? Well, we’ve already helped with this using our “Metrics Cards” in Bites. These are a set of Metrics well-defined with benchmarks to help you make fast decisions. Also, in the “Experiment Cards”, you’ll see suggested Metrics to track.
<aside> 👉 Access 50+ Bites here: www.febe.io/bites/home Customer who bought the StartHack Framework are automatically granted a full one year FREE subscription.
Simply use the discount code sent to your email when you signed up for StartHack in your checkout or reach out to our support [email protected]
</aside>
After picking the name of the KPI, let’s say [example], it’s time to now provide measure description by writing: “To verify that, we’ll measure [OMTM name]”
Then, we’ll define what success must look like and what failure must look like and write that down as an accountable documented way to call a hypothesis “validated” or “Invalidated”
Some OMTMs thought can have leading KPIs that can tell a more clear story. That’s what we call “Associated Metrics” that precedes or leads to the OMTM. This is optional but also highly recommended to at least have one. Simply, ask yourself what KPI I can define that can give me an early indicating signal to the OMTM.
Use your imagination to draft a hand sketch to easily communicate how you plan to implement your experiment.
It useless if you can’t find testers for your experiment. Simply start by writing down an overall description of your targets audience and then list down a min of 3 attributes to identify and find those audience.
Attributes are distinguishable characteristics that will help you to validate whether anyone fits your target audience description or not. We talk about it in details here.
Now that you have a clear picture about who your audience with, using the attribute, identify 3 channels or places you think you can find those audience. These channels can be think like Facebook, ProductHunt, search engine, etc. We talk more about channels here.
After listing those channels, go ahead and rank the best want to use for this experiment using the below factors:
Also for each channel, make sure to share a simple short line to explain how you’re intending to use this channel to reach out to your audience under the “strategy” column
Now that the plan is clear, list down all the needed tasks and actions to get it done ensuring to be productive by mentioning the day and time on which this needs to be done. This helps translates our plan into actionable steps to get things done
In the 'Analyze Results' section, input the results of the test, including the duration of the test and the number of customers involved. This is where you review the metrics you set before running the test (referencing the first Experiment Card for metric calculation methods) and analyze the data you've collected to determine the success or failure of your hypothesis.
Under 'Test Aftermath', compare the 'ESTIMATED' versus 'ACTUAL' costs incurred during the test, both for engineering (Eng Cost) and marketing (MKT Cost). This helps you understand the financial impact of the test and any discrepancies between your projections and the real-world expenses.
Title: Investigating Cost Discrepancies
The 'Why do you think it varies?' field is designed for reflection. If there were variances between your estimated and actual costs, articulate why these differences occurred. This could include unforeseen challenges, changes in scope, or any other factors that impacted the cost.
<aside> 👉 Use the LEARN section in the Hypothesis Canvas to document & reflect on your findings.
</aside>
After completing the Experiment Card, update your Validation Progress Board with the new information. This will help you keep track of which assumptions have been tested and validated and which ones need further exploration.
By thoroughly completing this post-experimentation process, startups can maintain a rigorous approach to testing their assumptions, ensuring that each test yields valuable insights that drive the business forward.