Idea Evaluation Framework

How Does It Work?

We collected nearly 90,000 product launches from the last two years and analyzed the feedback each of them received. More concretely, we quantified the use and buy intents within each comment or review, so we can aggregate these signals on the product level. This wasn't trivial (or cheap) because this meant we had to analyze 600,000 comments, totalling more than 50,000 pages.

Once we had all this data, we put it in a database so you can search it with any idea of your own and find similar products based on semantic similarity. The use and buy signals of those related, similar products help us predict the likely feedback your idea would receive.

Key metrics

To evaluate any idea we need to work out four key metrics for your idea.

Based on the first two, we can place your idea in a 3x3 matrix.

Confidence vs Engagement

Here's how we define the levels for each of dimension:
Confidence: Number of similar products.

• Low: 1 - 2
• Medium: 3 - 5
• High: >5

Engagement: Average number of comments across the similar products.

• Low: 1 - 4
• Medium: 4 - 10
• High: >10

These levels were defined by a healthy a mix of common-sense, trial & error and looking at our product launch data's key percentiles. Putting it simply, as we move from 1 to 9 in the figure above, we have more and more data to assess the use and buy signals of these similar products. Also note, that as confidence rises, so does your competition.

Adding the use & buy signals

We haven't yet added the use & buy signals to our framework. These are the key metrics that will help us determine if a product idea is worth pursuing. So let's spend a minute defining these more concretely.

Net use signal: Is defined as the difference between the positive and negative use signals. For example, if a 30% of a product's commenters explicitly said they would use it, and 5% said they would not, then the net use signal is 25%.

• Positive: More people would use the product than not.
• Negative: More people would not use the product than would.

Net buy signal: Defined similarly to the net use signal, as the difference between the positive and negative buy signals. For example, if 10% of a product's commenters explicitly said they would buy it, and 20% said they would not, then the net buy signal is -10%.

• Positive: More people would buy the product than not.
• Negative: More people would not buy the product than would.

When we put this into a 2x2 matrix we get four different scenarios.

Use vs Buy signal

1. This isn't good: People don't want to use or buy this product.
2. This is strange: People would buy this product but not use it (???).
3. This is promising: People want to use it but don't want to pay for it directly, e.g. freemium products.
4. This is what we want: People want to use it and are willing to pay for it too.

* Keep in mind, that just like we had to take the average of the number of comments across similar products to arrive at the overall engagement, we'll have to do the same for net use and net buy signals. Remember, each product has these two metrics, so we need to average them across all similar products to get net use and net buy signals for your idea.
** Actually, we use a weighted average where the weights are the relevance of the similar products to your idea.

Should I Build It Framework

As you can see from the figure below, we have a relatively simple framework, that let's you place any idea into one of ten types.

Should I Build It Idea Evaluation Framework

Let's walk through the ten idea types and define their key characteristics. We'll go from least interesting to most exciting.

1. No signal: You're exploring uncharted territory with no comparable products. While this could mean you've found a unique opportunity, it's more likely that either the problem isn't pressing enough or others have tried and failed silently. Proceed with extreme caution.
2. Minimal signal (MS): There's barely any market activity - either because the problem is very niche or not important enough. You'll need to prove real demand exists before investing significant time.
3. Mixed signal: People are willing to pay for similar solutions but don't enjoy using them. This suggests you might be solving a real problem in a way that frustrates users. The opportunity exists but requires careful refinement.
4. Swamp: The market is crowded with mediocre solutions that nobody loves or cares about. Unless you can offer something fundamentally different, you'll likely struggle to stand out or make money.
5. Run away: Multiple attempts have failed with clear negative feedback. Continuing down this path would likely waste your time and resources when better opportunities exist elsewhere.
6. Pivot: Current solutions aren't working well, but there might be a way to adjust your approach. This isn't about starting over, but rather making thoughtful changes based on what you're learning.
7. Competitive terrain: While there's clear interest in your idea, the market is saturated with similar offerings. To succeed, your product needs to stand out by offering something unique that competitors aren't providing. The challenge here isn't whether there's demand, but how you can capture attention and keep it.
8. Freemium: People love using similar products but resist paying. You'll need to either find who will pay or create additional value that's worth paying for.
9. Strong contender: The market has shown clear demand for this type of solution. Your challenge now is to create a version that stands out while delivering what people already want.
10. Early niche: A small but engaged group cares enough to comment, suggesting real potential if you can improve the experience. This is your chance to create something better for people who clearly want a solution.

A few notes:

• Each idea type is unique and requires a distinctive approach to handle it successfully. Therefore, we first categorize your idea into one of these 10 types, so we can create a highly personalized report for you, with actionable recommendations, and though provoking questions, tailored to your specific situation.
• "Early niche" is our personal favorite as bootstrapped technical founders as it represents the earliest form of an idea that might make it in the market without having to compete with too many previously launched products.
• "Strong contender" is also pretty good, but often you'll be up against one or two relatively well-established products with loyal customers.
• The more you move to the right (higher confidence) the stronger your competition will become. With higher number of similar products, you can be more certain of the correctness of the use and buy signals, but that's only possible because there are already a number of teams well ahead of you.
• There's a nice transition from "Pivot" to "Run away" in the bottom left quadrants (negative use & negative buy signal). This is because as we find more and more similar products, with the same double-negative signal, the less likely it becomes that you can pivot your way out of the situation.
• The "Swamp" category is probably worse than "Run away" because it can lure you into believing you're working on something that's got a chance, when in fact that fantasy is only kept alive by the low engagement of your users (which you fail to take seriously).
• We lump together all "Freemium" types into one, but the truth is, these can be very different in terms of competition and engagement.

Conclusion

We created a system to evaluate startup ideas, based on the feedback of 90k previous product launches. We didn't just build our system into a neat product that you can use, but also condensed it into a useful mental framework, that allows you to reason about and rank various startup or product ideas.

We know, that by doing so we came dangerously close to Conjoined Triangle Of Success territory. The irony is not lost on us. However, we hope that our framework is actually useful and will help you make better decisions with your new venture(s).

Almost certainly, we didn't get it right for the first time. So if you have any thoughts or feedback about this, please drop us a line at hello@shouldibuild.it. We would love to hear from you.