Strategy and Testing Business Ideas

In a series of previous posts, we talked about the need to create a set of tools that helps teams find objective ways to value design solutions. We’ve also looked at the different discussions that need to be facilitated while quantifying and qualifying strategy, namely: Pivot and Risk Mitigation, Facilitating Investment Discussions, and Visibility and Traceability.

In this post, I’ll talk about the importance of finding objective ways for exploring and (preferably) testing business ideas early – even before designs – to inform if they are worth pursuing (or not).

TL;DR;

• While in the past, designers would concentrate on enhancing desirability, the emerging strategic role of designers means they have to balance desirability, feasibility, and viability simultaneously. Designers need to expand their profiles and master a whole new set of strategic practices.
• Don’t make the mistake of executing business ideas without evidence; test your ideas thoroughly, regardless of how great they may seem in theory.
• Measurement allows the comparison of expected outcomes with actual outcomes and enables you to adjust strategic choices accordingly.
• Without learning, you risk delivering a product or service no one finds valuable.
• Your initial product strategy may contain plenty of assumptions and risks, and you may as well discover that the strategy is wrong and does not work.
• Experiments replace guesswork, intuition, and best practices with knowledge, which can help teams to make increasingly better decisions.
• When creating experiments, you don’t want to simulate any more than you need to. This is what allows you to integrate quickly through several assumption tests.
• Experiments will provide you with the data that will help you decide if you should persevere, pivot, or stop. Pivoting is attractive only if you pivot early when the cost of changing direction is comparatively low.

Quantifying and Qualifying Strategy

In a previous article, I mentioned that we need objective ways to value design solutions to justify the experience investments and look at the different points in the strategic planning and execution and identify the discussions that strategists should facilitate around what customers and users perceive as value while tracking and tracing the implementation of a strategy to ensure we are bringing value for both customers and business.

Design is the activity of turning vague ideas, market insights, and evidence into concrete value propositions and solid business models. Good design involves the use of strong business model patterns to maximize returns and compete beyond product, price and technology.

Bland, D. J., & Osterwalder, A., Testing business ideas, (2020)

From that perspective, we need to find ways to:

• Explore (and preferably test) ideas early
• Facilitate investment discussions by objectively describing business and user value, establishing priorities
• Asses risks of pursuing ideas while capturing signals that indicate if/when to pivot if an idea “doesn’t work”
• Capture and track the progress of strategy implementation

In that previous article, I went deep into quantification and metrics, so I suggest taking a look at that if you’re interested in measuring experiences.

Validating and Testing Business Ideas

“What do people need?” is a critical question to ask when you build a product. Wasting your life’s savings and your investor’s money, risking your reputation, making false promises to employees and potential partners, and trashing months of work you can never get back is a shame. It’s also a shame to find out you were completely delusional when you thought everyone needed the product you were working on (Sharon, T., Validating Product Ideas, 2016).

Don’t make the mistake of executing business ideas without evidence; test your ideas throughly, regardless of how great they may seem in theory.

Bland, D. J., & Osterwalder, A., Testing Business Ideas: A Field Guide for Rapid Experimentation, 2019).

To test a big business idea, you break it down into smaller chunks of testable hypotheses. These hypotheses cover three types of risk that (Bland, D. J., & Osterwalder, Testing Business Ideas: A Field Guide for Rapid Experimentation, 2019):

• First, customers aren’t interested in your idea (desirability).
• Second, you can’t build and deliver your idea (feasibility).
• Third, you can’t earn enough money from your idea (viability).

Facilitating Investment Discussions around Value

As I mentioned in a previous post, designers must become skilled facilitators that respond, prod, encourage, guide, coach, and teach as they guide individuals and groups to make decisions that are critical in the business world through effective processes. Few decisions are harder than deciding how to prioritize.

I’ve seen too many teams that a lot of their decisions seem to be driven by the question “What can we implement with the least effort” or “What are we able to implement” not by the question “what brings value to the user.”

From a user-centered perspective, the most crucial pivot that needs to happen in the conversation between designers and business stakeholders is the framing of value:

• Business value
• User value
• Value to designers (sense of self-realization? Did I positively impact someone’s life?)

The mistake I’ve seen many designers make is to look at prioritization discussion as a zero-sum game: our user-centered design tools set may have focused too much on the needs of the user at the expense of business needs and technological constraints.

While in the past designers would concentrate on enhancing desirability, the emerging strategic role of designers means they have to balance desirability, feasibility and viability simultaneously. Designers need to expand their profiles and master a whole new set of strategic practices.”

“Strategic Designers: Capital T-shaped professionals” in Strategic Design (Calabretta et al., 2016)

Experiments are a valuable tool for improving companies’ performance and facilitating better decision-making. When incentives are aligned between a company and its customers or employees, experiments can create a lot of value for all parties involved (Luca, M., & Bazerman, M. H., The power of experiments, 2021).

But there are also situations in which the value to customers or employees seems less clear, raising the potential for companies to gain financially at the expense of others. In those situations, it is up to companies to decide how to weigh financial profit against other factors, such as company reputation and customer satisfaction.

Luca, M., & Bazerman, M. H., The power of experiments (2021)

To understand the risk and uncertainty of your idea, you need to ask: “What are all the things that need to be true for this idea to work?” This will allow you to identify all three types of hypotheses underlying a business idea: desirability, feasibility, and viability (Bland, D. J., & Osterwalder, A., Testing business ideas, 2020):

• Desirability (do they want this?) relates to the risk that the market a business is targeting is too small; that too few customers want the value proposition; or that the company can’t reach, acquire, and retain targeted customers.
• Feasibility (Can we do this?) relates to the risk that a business can’t manage, scale, or get access to key resources (technology, IP, brand, etc.). This isn’t just technical feasibility; we also look need to look at overall regulations, policy, and governance that would prevent you from making your solution a success.
• Viability (Should we do this?) relates to the risk that a business cannot generate more revenue than costs (revenue stream and cost stream). While customers may want your solution (desirable), and you can build it (feasible), perhaps there’s not enough of a market for it, or people won’t pay enough for it. 

Design strategists should help the team find objective ways to value design ideas/ approaches/ solutions to justify the investment in them from both desirability, feasibility, and viability.

Forming Hypothesis

Every project begins with assumptions. There’s no getting around this fact. We assume we know our customers (and who our future customers will be). We assume we know what the competition is doing and where our industry is headed. We assume we can price the stability of our markets. These assumptions are predicated on our ability to predict the future. (Gothelf, J., & Seiden, J., Sense and respond, 2017).

If we accept that we’re always starting with assumptions, the real question becomes, What do we do about the risk of being wrong?

Gothelf, J., & Seiden, J., Sense and respond (2017).

Many companies try to deal with complexity with analytical firepower and sophisticated mathematics. That is unfortunate since the most essential elements of creating a hypothesis can typically be communicated through simple pencil-and-paper sketches (Govindarajan, V., & Trimble, C., The other side of innovation: Solving the execution challenge, 2010.)

The key to dealing with complexity is to focus on having good conversations about assumptions.

Break Down the Hypothesis in The other side of innovation: Solving the execution challenge, Govindarajan, V., & Trimble, C., (2010)

That said, flawed assumptions are one of the worst barriers to innovation. They’re invisible, chronic, and insidious, and we’re all ruled by them in one situation or another. How do they hold us back? (Griffiths, C., & Costi, M., The Creative Thinking Handbook: Your step-by-step guide to problem solving in business, 2019):

• They lead us to think we know all the facts when we don’t. Assumptions such as We have to launch a new range of products every year to keep up with competitors’ should be checked for validity.
• They cause us to become trapped by our own self-imposed limits and specializations, for example, Xerox’s failure to capture the personal computing market by limiting itself to making better copiers.
• Rules, like assumptions, keep us stuck in outdated patterns. The more entrenched the rule is, the greater the chance it’s no longer valid. Sometimes, we need to shake up or reverse our existing patterns to stand out from everyone else.

Usually, we want to start with the biggest questions and work our way down into the details. Typically, you would start with questions like these (Gothelf, J., & Seiden, J., Sense and respond, 2017):

• Does the business problem exist?
• Does the customer need exist?
• How do we know whether this feature or service will address that need?

As you sit down with your teams to plan out your next initiative, ask them these questions (Gothelf, J., & Seiden, J., Sense and respond. 2017):

• What is the most important thing (or things) we need to learn first?
• What is the fastest, most efficient way to learn that?

Prioritizing Hypotheses

If you only have one hypothesis to test, it’s clear where to spend the time you have to do discovery work. If you have many hypotheses, how do you decide where your precious discovery hours should be spent? Which hypotheses should be tested? Which ones should be de-prioritized or just thrown away? To help answer this question, Jeff Gothelf put together the Hypothesis Prioritisation Canvas (Gothelf, J., The hypothesis prioritization canvas, 2019):

Desirability Hypothesis

When assessing a Business Idea, we must always start by assessing Desirability because a failure to address their customers’ needs and find a product-market fit with a solution is the first hurdle that most business ideas fail to get past (Wong, R., Lean business scorecard: Desirability, 2021).

There are a lot of myths in the industry that seems to get in the way of objectively discussing what customers want. And while there is a degree of truth that “people cannot tell you want they want,” quantifying desirability is not impossible. We’ll get back to that when we cover qualitative methods for capturing preference data.

Feasibility Hypothesis

Maybe I’m an idealist, but I believe everything is feasible — given enough time and resources. The task of strategists then becomes understanding the expectations of stakeholders, facilitating the discussions necessary to identify the gap between vision and the current state, then working out what needs to be true to get to that vision.

With that being said, the gap between the current state and vision can only be filled by the people that are going to do the work, which is why I think a lot of projects fail: if decisions are made (e.g., roadmaps, release plans, investment priorities, etc.) without involving those people that actually going to do the work.

If the first time your developers see an idea is at the sprint planning, you have failed.

Cagan, M., Inspired: How to create tech products customers love (2017)

We need to ensure feasibility before we decide, not after. Not only does this end up saving a lot of wasted time, but it turns out that getting the engineers’ perspective earlier also tends to improve the solution itself, and it’s critical to shared learning (Cagan, M., Inspired: How to create tech products customers love, 2017).

Feasibility focuses on the business operations and capabilities needed to run and grow a business idea, not just the feasibility of the solution at the core of the idea

Wong, R., Lean business scorecard: Feasibility (2021)

If you have taken the necessary steps to focus first on validating the Desirability and Viability of a business, you should have a clear understanding of what your killer feature is, and you should know why it’s so valuable for your customers (Wong, R., Lean business scorecard: Feasibility, 2021):

• Your Key Partners — the people and organizations who are essential to providing your business with the leanest set of capabilities to run your business and drive growth. If you’re thinking of flying solo on a business venture, consider again your limited time and what you choose to do with it. You will need to make decisions and trade-offs on what to buy, borrow or build to operate your business, and it may not always make sense to have all your capabilities in-house from day one or ever.
• Key Capabilities — the capabilities you need to ideate, create, release and operate. Typically capabilities are things that allow a business to do something and can be in the form of people, processes, information, or technology.
• Key Activities — what you do on a day-to-day basis with all your capabilities to drive growth and win your customers’ business

Viability Hypothesis

Similarly to Feasibility, we need to validate the business viability of our ideas during discovery, not after (Cagan, M., Inspired: How to create tech products customers love, 2017).

It’s absolutely critical to ensure that the solution we build will meet the needs of our business — before we talk the time and expense to build out the product.

Cagan, M., Inspired: How to create tech products customers love (2017)

Once you have sufficient evidence that you’ve found the right opportunity to address AND you have a solution that helps your target audience do something they couldn’t before, you then need to prove you can get paid enough for this product or service to have a commercially viable business that can sustain itself over time (Wong, R., Lean business scorecard: Viability, 2021).

If we’re going to do the hard work to realize the full value of a business, I like to know that it’s worth putting the time, energy, and potential money into the venture. It’s good to set your sights on a prize worth winning in the end (Wong, R., Lean business scorecard: Viability, 2021):

• Have we got a large enough group of people we can do business with? A group that has the same job to be done and challenges that we identified in the previous section. A group that may be served by existing competitors that we may need to entice away.
• How much value will that group of people provide to us? When starting out, it’s often useful to validate a business idea by seeing if someone is willing to give you something small like their email address in exchange for something they want before asking them for money. Does this idea have any evidence that someone has exchanged something of value from them in return for a product or service? By definition, something has the potential for Viability if a value exchange takes place, but whether it’s profitable or sustainable for a business is a whole other question.
• What does the customer get in exchange? Another way to do something relatively simple to test the Viability of your business idea is to set up a website landing page that promises some sort of valuable product or service in the future in return for an email address now. This is a good way to test viability on a small scale, as the value exchange is small for everyone. But as the perceived value of a product or service increases, so should the willingness to pay and the willingness to work harder to get that value.

Quantitative Aspects of Testing Business Ideas

It’s an old saying that says what gets measured gets done. There is more than a little truth to this. If aspirations are to be achieved, capabilities developed, and management systems created, progress needs to be measured (“Manage What Matters” in Playing to Win: How Strategy Really Works (Lafley, A.G., Martin, R. L., 2013).

Measurement allows comparison of expected outcomes with actual outcomes and enables you to adjust strategic choices accordingly.

“Manage What Matters” in Playing to Win: How Strategy Really Works (Lafley, A.G., Martin, R. L., 2013)

I will refrain from proposing a single metric for quantifying and qualifying design for a few reasons:

• Different organizations have specific strategic choices about winning that uniquely positions in their corresponding industry: these metrics should take into consideration both the goals of users, but also what the business is trying to learn from the study, then design usability studies accordingly.
• Different organizations are different levels of design maturity: if you’ve never done any usability studies, it’s not only hard to build the capability to run such studies but also figure out how to feedback the information back into product decisions.
• Some of these metrics are discovered too late: since some of these metrics are collected either during usability studies, or after the product or service is released, it means that — by the time you collect them — a lot of product decisions have already been made. Some of these decisions could be quite expensive to reverse or pivot at that point, so it might be too late for quantifying and qualifying the success of the strategy.
• Beware of how you measure: quantitative metrics are good to explain ‘What’ and ‘How many’ of a given hypothesis; the ‘Why’s are usually better captured through qualitative research methods
• The world is different now: some of the signals and indicators that worked for measuring success may not work for new products or services you are trying to create.

Never assume that the metrics and standards used to evaluate the existing business have relevance for the innovation initiative.

Govindarajan, V., & Trimble, C., The other side of innovation: Solving the execution challenge (2010)

Quantifying and Qualifying the value and performance of design does not need to be complex or foreboding. There is a case for intuition, a case for qualitative user research, a case for quantitative research, and a case for synthesis. And there is even room for imponderables because some things are simply beyond definition or measurement (Lockwood, T., “Design Value: A Framework for Measurement” in Building Design Strategy, 2008).

Quantitative research can tell you how many customers are doing (or not doing) something. But it won’t tell you why the customers are doing it (or not doing it).

Olsen, D. The lean product playbook, 2015

So, what would a set of tools that both empowers intuition and creativity but also help us find objective ways to value design solutions look like?

Pirate Metrics (a.k.a. AARRR!)

Pirate Metrics—a term coined by venture capitalist Dave McClure—gets its name from the acronym for five distinct elements of building a successful business. McClure categorizes the metrics a startup needs to watch into acquisition, activation, retention, revenue, and referral—AARRR (Croll, A., & Yoskovitz, B. Lean Analytics. 2013).

McClure recommends tracking two or three key metrics for each of the five elements of his framework. That is a good idea because your conversion funnel isn’t just one overall metric; you can track the more details metrics, making a distinction between the macro-metrics and the micro-metrics that relate to them (Olsen, D. The lean product playbook, 2015).

HEART Metrics

What the research team from Google noted was that while small-scale frameworks were commonplace, measuring the experience on a large scale via automated means had no framework in place. Thus the Heart Framework is specifically targeted at that kind of measurement. However, the principles are equally useful at a small scale level, though the methodologies used to derive measurements at a smaller scale are likely to be substantially different (Rodden, K., Hutchinson, H., & Fu, X., Measuring the user experience on a large scale: User-centered metrics for web applications, 2010).

There are five metrics used in the HEART framework:

• Happiness
• Engagement
• Adoption
• Retention
• Task Success

HEART can be seen as a manifestation of the Technology Acceptance Model (TAM)—after all, both include Adoption in their names. The TAM is itself a manifestation of the Theory of Reasoned Action. The TRA is a model that predicts behavior from attitudes. The TAM suggests that people will adopt and continue to use technology (the EAR of the HEART) based on the perception of how easy it is to use (H), how easy it is to use (T), and whether it’s perceived as useful (H). The SUS, SEQ, and the ease components of the UMUX-Lite and SUPR-Q are all great examples of measuring perceived ease, and bringing Happiness to the HEART model (Sauro, J., Should you love the HEART framework?, 2019):

North Star Metrics

A north star metric is a key performance indicator (KPI) that you use to measure the progress of your business. It has one purpose: to keep you focused on what’s important. A metric shouldn’t be something obscure or abstract, like “more customers” or “higher engagement.” Those goals can be helpful and can be used as input into your north star metric, but they don’t make great KPIs themselves because they don’t provide any information about how well you’re meeting them (Gadvi, V., How to identify your North Star Metric, 2022).

A north star metric is one metric that captures the core value that your product delivers to customers, ideally in a single number. It should be specific to your business and easy to understand and measure.

Gadvi, V., How to identify your North Star Metric (2022)

Let’s look at four reasons why you should use North Star, or “the One Metric That Matters” OMTM (Croll, A., & Yoskovitz, B. Lean Analytics. 2013):

• It answers the most important question you have. At any given time, you’ll be trying to answer a hundred different questions and juggling a million things. You need to identify the riskiest areas of your business as quickly as possible, and that’s where the most important question lies. When you know what the right question is, you’ll know what metric to track in order to answer that question. That’s the OMTM.
• It forces you to draw a line in the sand and have clear goals. After you’ve identified the key problem on which you want to focus, you need to set goals. You need a way of defining success.
• It focuses the entire company. Avinash Kaushik has a name for trying to report too many things: data puking. Nobody likes puke. Use the OMTM as a way of focusing your entire company. Display your OMTM prominently through web dashboards, on TV screens, or in regular emails.
• It inspires a culture of experimentation. By now, you should appreciate the importance of experimentation. It’s critical to move through the build-›measure-*learn cycle as quickly and as frequently as possible. To succeed at that, you need to encourage experimentation actively. It will lead to small-f failures, but you can’t punish that. Quite the opposite: failure that comes from planned, methodical testing is how you learn. It moves things forward in the end. It’s how you avoid the big-F failure. Everyone in your organization should be inspired and encouraged to experiment. When everyone rallies around the One Metric that Matters, it gives us the opportunity to experiment independently to improve it, it’s a powerful force.

Product managers working in established companies have this figured out, but if you’re a founding product manager or an entrepreneur, here’s what it means for you. The key to picking the right North Start / OMTM metrics is to find the one that appropriately aligns with your business model (check the table below). So, let’s say you were the founder of an online store selling vegan products. Your North Star Metric would be the Average Order Value – defined as the total amount spent per order over a specific period. It is calculated using the following formula (Gadvi, V., How to identify your North Star Metric, 2022):

Business Model	Example	North Star Metrics
User Generated Content + Ads	Facebook, Quora, Instagram, Youtube	Monthly Active Users (MAU), Time on Site(ToS)
Freemium	Spotify, Mobile Games, Tinder	Monthly Active Users (MAU), % who upgrade to paid
Enterprise SAAS	Slack, Asana	Monthly Active Users (MAU), % who upgrade to paid
2-sided marketplace	Airbnb, Uber	Monthly Active Users (MAU), monthly active riders/drivers, Monthly Active Users (MAU) buyers/sellers
Ecommerce	Amazon, eBay, Flipkart	Average order value (AOV); basket size

Defining a North Star metric, and evaluating each release in relation to that metric, will help drive your business forward. In early stages of the product lifecycle, your metric will be more qualitative-but if it’s indicative of customer value, it will tell you whether you’re headed in the right direction.

Garbugli, É.  Solving Product, (2020)

Your north star metric should also be accessible for all your team members to understand and communicate, even if they don’t work in data science or analytics. Having a clear north star metric helps everyone in the organization stay aligned around what matters most when making decisions about new features or products — which will ultimately make them more successful by bringing them closer to their users’ needs (Gadvi, V., How to identify your North Star Metric, 2022).

Each strategy we had at Netflix (from our personalization strategy to our theory that a simpler experience would improve retention) had a very specific metric that helped us to evaluate if the strategy was valid or not. If the strategy moved the metric, we knew we were on the right path. If we failed to move the metric, we moved on the next idea. Identifying these metrics took a lot of the politics and ambiguity out of which strategies were succeeding or not.Gibson Biddle, Netflix Former VP of Product in Solving Product (Garbugli, É., 2020)

Qualitative Aspects of Testing Business Ideas

As I mentioned early in this article, quantitative data may be easier to get, but it has its limitations:

• Some of these metrics are discovered too late: since some of these metrics are collected either during usability studies or after the product or service is released, it means that — by the time you collect them — a lot of product decisions have already been made. Some of these decisions could be quite expensive to reverse or pivot at that point, so it might be too late for quantifying and qualifying the success of the strategy.
• Beware of how you measure: quantitative metrics are good to explain ‘What’ and ‘How many’ of a given hypothesis; the ‘Why’s are usually better captured through qualitative research methods

As I mentioned above, there are a lot of myths in the industry that seems to get in the way of objectively discussing what customers want. And while there is a degree of truth that “people cannot tell you want they want,” quantifying desirability is not impossible. Let’s talk about some methods now!

Quantifying and Qualifying Value, Satisfaction and Desirability

When product managers, designers, and strategists are crafting their strategy or working on the discovery phase, the kind of user and customer insights they are looking for is really hard to acquire through quantitative metrics, either because we cannot derive insights from the existing analytics coming from the product, or because we are creating something new (so there are no numbers to refer to). Most of such insights (especially desirability and satisfaction) would come from preference data.

Preference data consists of the more subjective data that measures a participant’s feelings or opinions of the product.

Rubin, J., & Chisnell, D., Handbook of usability testing: How to plan, design, and conduct effective tests (2011)

Just because preference data is more subjective, it doesn’t mean is less quantifiable: although the design and several usability activities are certainly qualitative, the image of good and bad designs can easily be quantified through metrics like perceived satisfaction, recommendations, etc. (Sauro, J., & Lewis, J. R., Quantifying the user experience: Practical statistics for user research. 2016).

Preference Data is typically collected via written, oral, or even online questionnaires or through the debriefing session of a test. A rating scale that measures how a participant feels about the product is an example of a preference measure (Rubin, J., & Chisnell, D., Handbook of usability testing, 2011).

You can find examples of preference data that design strategists can collect to inform strategic decisions in my previous post, so I’ll just mention a few here:

Value Opportunity Analysis (VOA)

Value Opportunity Analysis (VOA) is an evaluative method that creates a measurable way to predict the success or failure of a product by focusing on the user’s point of view. The Value Opportunity Analysis (VOA) can happen at two stages throughout the design process (Hanington, B., & Martin, B., Universal methods of design, 2012):

• VOA is typically used in the concept generation stage when prototyping is still low fidelity or even on paper.
• It is also used at the launch stage and tested for quality assurance to determine market readiness.  An example could be testing a current design prior to investing in a redesign.

There are seven value opportunities (Hanington, B., & Martin, B., Universal methods of design, 2012):

1. Emotion: Adventure, Independence, Security, Sensuality, Confidence, Power
2. Aesthetics: Visual, Auditory, Tactile, Olfactory, Taste
3. Identity: Point in time, Sense of Place, Personality
4. Impact: Social, Environmental
5. Ergonomics: Comfort, Safety, Ease of use
6. Core Technology: Reliable, Enabling
7. Quality: Craftsmanship, Durability

Usefulness, Satisfaction, and Ease of Use (USE)

The Usefulness, Satisfaction, and Ease of Use Questionnaire (USE, Lund, 2001) measures the subjective usability of a product or service. It is a 30-item survey that examines four dimensions of usability (Sauro, J., & Lewis, J. R., Quantifying the user experience. 2016):

• Usefulness
• Ease of use
• Ease of learning
• Satisfaction.

American Customer Satisfaction Index (ACSI)

The Satisfaction level indicator is reliant on 3 critical 10-point scale questions to obtain customer satisfaction. These American Customer Satisfaction Index (ACSI) questions are categorized into (Fornell et al, The American Customer Satisfaction Index, 1996):

• Satisfaction
• Expectation Levels
• Performance

While intended as a macroeconomic measure of U.S. consumers in general, many corporations have used the American Customer Satisfaction Index (ACSI) to quantify and qualify the satisfaction of their own customers.

System Usability Scale (SUS)

The System Usability Scale (SUS) consists of ten statements to which participants rate their level of agreement. No attempt is made to assess different attributes of the system (e.g. usability, usefulness, etc.): the intent is to look at the combined rating (Tullis, T., & Albert, W., Measuring the user experience. 2013).

Usability Metric for User Experience (UMUX)

In response to the need for a shorter questionnaire, Finstad introduced the Usability Metric for User Experience (UMUX) in 2010. It’s intended to be similar to the SUS but is shorter and targeted toward the ISO 9241 definition of usability (effectiveness, efficiency, and satisfaction). It contains two positive and two negative items with a 7-point response scale. The four items are (Sauro, J., & Lewis, J. R., Quantifying the user experience. 2016):

• [This system’s] capabilities meet my requirements.
• Using [this system] is a frustrating experience.
• [This system] is easy to use.
• I have to spend too much time correcting things with [this system].

UMUX-Lite

To improve the UMUX, Lewis et al (“Measuring perceived usability: The SUS, UMUX-LITE, and AltUsability” in International Journal of Human-Computer Interaction, 31(8), 496–505, 2015) proposed a shorter all-positive questionnaire called the UMUX-Lite using the same 7-point scale with the following two items (Sauro, J., & Lewis, J. R., Quantifying the user experience. 2016):

• [This system’s] capabilities meet my requirements.
• [This system] is easy to use.

Desirability Testing

A Desirability Test is great for gauging first-impression emotional responses to products and services (Hanington, B., & Martin, B., Universal methods of design, 2012):

• Explores the affective responses that different designs elicit from people based on first impressions.
• Using index cards with positive, neutral, and negative adjectives written on them, participants pick those that describe how they feel about a design or a prototype.
• Can be conducted using low-fidelity prototypes as a baseline before the team embarks on a redesign.

Participants are offered different visual-design alternatives and are expected to associate each alternative with a set of attributes selected from a closed list.

Jobs To Be Done (JTBD) and Outcome-Driven Innovation

Outcome-Driven Innovation (ODI) is a strategy and innovation process built around the theory that people buy products and services to get jobs done. It links a company’s value creation activities to quantifying and qualifying customer-defined metrics. Ulwick found that previous innovation practices were ineffective because they were incomplete, overlapping, or unnecessary.

Outcome-Driven Innovation® (ODI) is a strategy and innovation process that enables a company to create and market winning product and service offerings with a success rate that is 5-times the industry average

Ulwick, A.,  What customers want: Using outcome-driven innovation to create breakthrough products and services (2005)

Clayton Christensen credits Ulwick and Richard Pedi of Gage Foods with the way of thinking about market structure used in the chapter “What Products Will Customers Want to Buy?” in his Innovator’s Solution and called “jobs to be done” or “outcomes that customers are seeking.”

Ulwick’s “opportunity algorithm” measures and ranks innovation opportunities. Standard gap analysis looks at the simple difference between importance and satisfaction metrics; Ulwick’s formula gives twice as much weight to importance as to satisfaction, where importance and satisfaction are the proportion of high survey responses.

You’re probably asking yourself, “where do these values come from?” That’s where User Research comes in handy: once you’ve got the List of Use Cases, you go back to your users and probe on how important each use case is and how satisfied with the product they are with regards to each use case.

Once you’ve obtained the opportunity scores for each use case, what comes next? There are two complementary pieces of information that the scores reveal: where the market is underserved and where it is overserved. We can use this information to make some important targeting and resource-related decisions.

Almost as important as knowing where the market is underserved is knowing where it is overserved. Jobs and outcomes that are unimportant or already satisfied represent little opportunity for improvement and consequently should not receive any resource allocation in most markets. It is not uncommon to find a number of outcomes that are overserved-and companies that are nevertheless continuing to allocate them development resources (Ulwick, A. W., What customers want, 2005).

Design and Conduct Tests

Experiments have generated large returns for tech companies. To
give just a few examples:

• Microsoft’s search engine, Bing, ran an experiment in which it varied the physical size of ads on the screen. Bing found that increasing the size of advertisements increased user engagement, even though users saw fewer total ads. This simple change generated an additional $50 million per year in profits, according to Ron Kohavi, a VP of experimentation at Microsoft’s Cloud and Al group.
• Drawing on economic theory, economists Michael Ostrovsky and Michael Schwarz ran an experiment at Yahoo in which they tested new rules on the auction system it used to sell advertisements and increased Yahoo’s profits by millions of dollars per year. (Ostrovsky, M., & Schwarz, M., 2011).
• Amazon found that moving credit card offers from the homepage to the shopping cart page increased profits by millions of dollars. (Kohavi, R., & Thomke, S., 2017).

In particular, experiments can benefit organizations in four main ways (Luca, M., & Bazerman, M. H., The power of experiments, 2021):

• Testing theory and mechanisms: in many situations, managers have a theory about what is happening in a certain situation. Running an experiment can help to confirm (or disconfirm) such theories and shed light on what social scientists refer to as mechanisms — in other words, what is driving the patterns we observe.
• Understanding magnitudes and tradeoffs: Experiments help clearly define the tradeoffs that are occurring as a result of design changes. Once teams have identified these tradeoffs, they can be assured that if they disagree about a path forward, it’s based on differences in tradeoff preferences rather than different guesses about what will work.
• Evaluating policies: Airbnb made a bunch of decisions about what bundle of changes to make. Get rid of host pictures from the search page? Check. Get rid of host pictures from the listing page? Nope. Take steps to increase instant booking? Yes. Require instant booking? No. At the end of all of this, the company rolled out its redesigned Platform. Because these changes affect one another, and because the company’s experiments may have tested different options than the final version, there was value not only in testing individual theories, mechanisms, and tweaks but also in evaluating the new page as a whole. This wouldn’t allow Airbnb to identify exactly which tweaks were driving different levels of change, as earlier experiments had, but it would show the company the overall impact of its suite of changes.
• Fact-finding: Sometimes you don’t have a theory. There are plenty of times when you just want to make sure you’re not breaking anything or missing some great innovation, or you just want to know what would happen if you tweaked a process. And that’s a fine use of experiments for searching out facts where you think the process might matter but don’t know exactly how. For example, suppose you were helping eBay to choose its font. Should they switch to Courier? It may be hard to have a good theory for this. But it’s easy just to check the results.

You test the most important hypothesis with appropriate experiments. Each experiment generates evidence and insights that allow you to learn and decide. Based on the evidence and your insights you either adapt your idea, if you learn that you were on the wrong path, or continue testing other aspects of your ideas if the evidence supports your direction (Bland, D. J., & Osterwalder, Testing Business Ideas: A Field Guide for Rapid Experimentation, 2019).

A successful project is not deemed successful because it is delivered accordant to a plan, but because it stood the test of reality.

“Walk the walk” in The decision maker’s playbook. Mueller, S., & Dhar, J. (2019)

By building, measuring, and learning, designers are able to get closer to great user experiences sooner rather than later (Gothelf, J., & Seiden, J., Lean UX: Applying lean principles to improve user experience, 2021)

Experiments and Outcomes

Experimentation is at the heart of what software developers call agile development. Rather than planning all activities up-front and then sequentially, agile development emphasizes running many experiments and learning from them (Mueller, S., & Dhar, J., The decision maker’s playbook, 2019)

Experiments replace guesswork, intuition and best practices with knowledge.

Mueller, S., & Dhar, J., The decision maker’s playbook (2019)

It takes a certain level of maturity to run effective experiments. To avoid shipping experiments for the sake of shipping experiments, teams need to focus on delivering outcomes. They also need to be willing to embrace failure to make progress (Garbugli, É., Solving Product, 2020).

With the right mindset, experiments can help teams to make increasingly better decisions.

Garbugli, É., Solving Product, 2020

To create this mindset, assemble a multidisciplinary team and let them work out their own process. Teresa Torres calls this multidisciplinary team “Product Trio” (a designer, a product manager, and a developer).

For a learning culture to thrive, your teams must feel safe to experiment. Experiments are how we learn, but experiments — by nature — fail frequently. In a good experiment, you learn as much from failure as from success. If failure is stigmatized, teams will take few risks (Gothelf, J., & Seiden, J., Sense and respond. 2017).

On average, 80% of experiments fail to deliver the expected outcomes but with the right method, 100% of experiments can help you learn and progress (Garbugli, É., Solving Product, 2020).

This means that your progress will not be linear and predictable and that you should not be judged by your delivery rate (the amount of stuff you ship) but by your learning rate, and by your overall progress towards strategic goals — in other words, by the extent to which you achieve the outcomes in question.

Gothelf, J., & Seiden, J., Sense and respond (2017).

You should focus on one or two core goals at a time, aligning with your North Star metric or the AARRR steps that you’re focused on. Your goals should be big, and your experiments small and nimble (Garbugli, É., Solving Product, 2020).

Teams will be more willing to experiment if they feel they are not being measured by the delivery of hard requirements but appreciated by achieving great outcomes that create value.

You might be asking, “what you do mean by outcome?”. Joshua Seiden defines as outcome “a change in user behavior that drives business results.”

Using outcomes creates focus and alignment. It eliminates needless work. And it puts the customer at the center of everything you do

Seiden, J., Outcomes over Output (2019)

You can help the team and leaders to start thinking in terms of outcomes by asking three simple questions (Seiden, J., Outcomes over Output, 2019):

• What are the user and customer behaviors that drive business results? If the team gets stuck in trying to answer that question, there is a good chance that working on alignment diagrams will help.
• How do we get people to do more of these things?
• How do we know we’re right? The easiest (and the hardest) way to answer that question is to design and conduct tests.

Managing by outcomes communicates to the team how they should be measuring success. A clear outcome helps a team align around the work they should be prioritizing, helps them choose the right customer opportunities to address, and helps them measure the impact of their experiments. Without a clear outcome, discovery work can be never-ending, fruitless, and frustrating (Torres, T., Continuous Discovery Habits, 2021).

Testing Business Ideas through Experiments

Technology is awesome. It really is. It helps humans communicate, find old friends, work more effectively, have fun, find places, and oh-so-many other great things. In many cases, technology is also hard, time-consuming, and expensive to develop. In this step, you will need to find a way to solve a problem you want to solve with or without technology. Manual ways of solving problems are, without a doubt, inefficient, yet they will teach you a lot about what people want without actually developing any technology (Sharon, T., Validating Product Ideas, 2016).

Learning through experimentation has a number of benefits (Mueller, S., & Dhar, J., The decision maker’s playbook, 2019):

• It allows you to focus on actual outcomes: a successful project is not deemed successful because it is delivered according to a plan, but because it stood the test of reality.
• It decreases re-work: because the feedback cycles are short, potential errors or problems are spotted quickly and can be smoothed out faster than conventional planning.
• It reduces risks: because of increased transparency throughout the implementation process, risks can be better managed than in conventional projects.

One way to help the team think through experiments is to think about how are we going to answer the following questions (Croll, A., & Yoskovitz, B. Lean Analytics. 2013):

• What do you want to learn and why?
• What is the underlying problem we are trying to solve, and who is feeling the pain? This helps everyone involved have empathy for what we are doing.
• What is our hypothesis?
• How will we run the experiment, and what will we build to support it?
• Is the experiment safe to run?
• How will we conclude the experiment, and what steps will be taken to mitigate issues that result from the experiment’s conclusion?
• What measures will we use to invalidate our hypothesis with data? Include what measures will indicate the experiment isn’t safe to continue.

What’s important to understand is that testing rarely means just building a smaller version of what you want to sell. It’s not about building, or selling something. It’s about testing the most important assumptions, to show this idea could work. And that does not necessarily require building anything for a very long time. You need to first prove that there’s a market, that people have jobs with real pains and gains, and that they’re willing to pay (Bland, D. J., & Osterwalder, A., Testing business ideas, 2020).

A strong assumption test simulates an experience, giving your participant the opportunity to behave either in accordance with your assumption or not. This behavior is what allows us to evaluate our assumptions (Torres, T., Continuous Discovery Habits, 2021).

To construct a good assumption test, you’ll want to think carefully about the right moment to simulate (Torres, T., Continuous Discovery Habits, 2021).

You don’t want to simulate any more than you need to. This is what allows you to integrate quickly through several assumption tests.

Torres, T., Continuous Discovery Habits (2021)

With any new business experiment, you need to ask yourself how quickly you can get started and how quickly it produces insights. For example, an interview series with potential customers or partners can be set up fairly quickly. Launching a landing page and driving traffic to it can be done with even greater speed. You’ll generate insights quickly. A technology prototype on the other hand will take far more time to design and test. Such a prototype might gather a good understanding of user behavior, but it will require more time to generate insights (Bland, D. J., & Osterwalder, A., Testing business ideas, 2020).

More and more organizations test their business ideas before implementing them. The best ones perform a mix of experiments to prove that their ideas have legs. They ask two fundamental questions to design the ideal mix of experiments:

1. Speed: How quickly does an experiment produce insights?
2. Strength: How strong is the evidence produced by an experiment?

Before jumping to an experiment, it is important to consider the key principles of rapid experimentation when testing a new business idea (Bland, D. J., How to select the next best test from the experiment library, 2020):

1.  Go cheap and fast early on in your journey – Don’t spend a lot of money, if possible early on. You are only beginning to understand the problem space, so you don’t want to spend money when you can learn for free. Also, try to move quickly to learn fast instead of slow and perfectly.
   
   Quite often, I am asked, “how many experiments should I run”? This is a very hard question to answer as it depends on so many variables. But a general rule of thumb is 12 experiments in 12 weeks. 1 experiment a week is a good pace to keep up the momentum, and 12 data points are typically a good checkpoint to come back and reassess your business model.
   
2. Increase the strength of evidence with multiple experiments for the same hypothesis – Don’t hesitate to run multiple experiments for the same hypothesis. Rarely do we witness a team that only runs one experiment and uncovers a multimillion-dollar opportunity. The idea here is to not get too excited or too depressed after running one experiment. Give yourself permission to run multiple to understand if you have genuinely validated your hypothesis.
   
   Remember, a typical business model may have several critical hypotheses you need to test, and for each hypothesis, you may need to run multiple experiments to validate it. Refer to our previous blog post on Assumptions Mapping to learn how to define your hypotheses and determine which ones to run first.
   
3. Always pick the experiment that produces the strongest evidence, given your constraints – Not every experiment applies to every business. B2B differs from B2C, which differs from B2G. A 100-year-old corporation’s brand is much more important than a 100-hour-old startup’s brand. Pick an experiment that produces evidence, but don’t risk it all. Make small bets that are safe to fail.
   
   When working with corporate innovators, the phrase I most often hear is, “we can’t do that.” Innovators often feel hamstrung when working in heavily regulated industries. Many may choose to bypass testing certain hypotheses because of the constraints. This is not something I would recommend. If you really cannot test a hypothesis, the last resort would be to consider pivoting your business model. Remember, a testable idea is always better than a good idea.
   
4. Reduce uncertainty as much as you can before you build anything – In this day and age, you can learn quite a bit without building anything at all. Deferring your build as long as possible because it is often the most expensive way to learn.

In the Experiment Library, David Bland compiled a list of creative ways to demonstrate a prototype without building out your final product. Experiments such as Wizard of Oz, Clickable Prototype, and Single Feature MVP

Discovery or Validation?

Pulling inspiration from Steve Blank’s 4 Steps to the Epiphany, it is obvious that David Bland similarly framed the experiment library. One of the biggest differences, however, is that instead of customer discovery and customer validation, he decided to test the entire business model. Discovery experiments are often open-ended and directional, whereas Validation experiments have more of a true value exchange. Most teams start with discovery since the validation experiments cost more and take longer (Bland, D. J., How to select the next best test from the experiment library, 2020)

Early Signs versus Large Scale Experiments

Inevitably, someone on your team is going to raise a concern about making decisions based on small numbers. How can we have confidence in the data if we talk to only five customers? You might be tempted to test with larger pools of people to help get buy-in. But this strategy comes at a cost—it takes more time. We don’t want to invest time, energy, and effort into an experiment if we don’t even have an early signal that we are on the right track (Torres, T., Continuous Discovery Habits, 2021)

Rather than starting with a large-scale experiment (e.g., surveying hundreds of customers, launching a production-quality A/ B test, worrying about representative samples), we want to start small. You’ll be pleasantly surprised by how much you can learn from getting feedback from a handful of customers.”

Torres, T., Continuous Discovery Habits (2021)

With assumption testing, most of our learning comes from failed tests. That’s when we learn that something we thought was true might not be. Small tests give us a chance to fail sooner. Failing faster is what allows us to quickly move on to the next assumption, idea, or opportunity (Torres, T., Continuous Discovery Habits, 2021).

As we test assumptions, we want to start small and iterate our way to bigger, more reliable, more sound tests, only after each previous round provides an indicator that continuing to invest is worth our effort. We stop testing when we’ve removed enough risk and/ or the effort to run the next test is so great that it makes more sense to simply build the idea.”

Torres, T., Continuous Discovery Habits (2021)

Karl Popper, a renowned 20th-century philosopher of science, in the opening quote argues, “Good tests kill flawed theories,” preventing us from investing where there is little reward, and “we remain alive to guess again,” giving us another chance to get it right (Torres, T., Continuous Discovery Habits, 2021).

Risk Analysis and Assessment

Once risks are identified, they can be prioritized according to their potential impact and the likelihood of them occurring. This helps to highlight not only where things might go wrong and what their impact would be but how, why, and where these catalysts might be triggered (Kourdi, J., Business Strategy: A guide to effective decision-making, 2015):

• Technology. New hardware, software, or system configuration can trigger risks, as can new demand on existing information systems and technology.
• Organisational Change. Risks are triggered by — for example — new management structures or reporting lines, new strategies, and commercial agreements.
• Processes. New products, markets, and acquisitions all cause change and can trigger risks.
• People. New employees, losing key people, poor succession planning, or weak people management can all create dislocation, but the main danger is behavior: everything from laziness to fraud, exhaustion, and simple human error can trigger risks.
• External factors. Changes in regulation and political, economic, or social developments can all affect strategic decisions by bringing to the surface risks that may have lain hidden.

Without learning, you risk delivering a product or service no one finds valuable.

Gothelf, J., & Seiden, J., “The Sense and Repond Model” in Sense and respond (2017)

Analyze risks at the start of each iteration (or test); reassess them regularly. For each identified risk, ask yourself (Podeswa, H., “Analyse Risk” in The Business Analyst’s Handbook. 2008):

• Who owns the risk?
• What is the likelihood of the risk occurring?
• What is the impact on the business if it occurs?
• What is the best strategy for dealing with this risk?
• Is there anything that can be done to prevent it from happening or to mitigate (lessen) the damage if it does occur?

Start by selecting the biggest risks: the uncertainty that must be addressed now so that you don’t take the product in the wrong direction and experience late failure (e.g., figuring out at a late stage that you are building a product nobody wants or needs). Next, determine how you can best address the risks — for instance, by observing target users, interviewing customers, or employing a minimum viable product (MVP). Carry out the necessary work and collect relevant feedback or data. Then analyze the results and use the newly gained insights to decide if you should persevere, pivot, or stop — if you should stick with your strategy, change it, or no longer pursue your vision and take the appropriate actions accordingly (Pichler, R., Strategize, 2016).

Iteratively reworking the product strategy encourages you to carry out just enough market research just in time to avoid too much or too little research, addressing the biggest risks first so that you can quickly understand which parts of your strategy are working and which are not, thus avoid late failure (Pichler, R., Strategize, 2016).

Learn and Decide

Learning faster than everyone else is no longer enough. You need to put that learning into action because what you’ve learned has an expiration date: markets and technology move so quickly that the insights you’ve gained can expire within months, weeks, or even days. You should take action, such as (Bland, D. J., & Osterwalder, A., Testing business ideas, 2020):

• Next steps to make progress with testing and de-risking a business idea.
• Informed decisions based on collected insights.
• Decisions to abandon, change, and/or continue testing a business idea.

Persevere, Pivot, or Stop

Once you have collected the relevant feedback or data, reviewed and analyzed it, ask yourself if your strategy is still valid: your initial product strategy may contain plenty of assumptions and risks, and you may as well discover that the strategy is wrong and does not work. If that is the case, then you have two choices (Pichler, R., Strategize, 2016):

• stop and let go of your vision, or
• stick with the vision and change the strategy, which is also called a pivot.

Pivoting is attractive only if you pivot early, when the cost of changing direction is comparatively low.

Pichler, R., Strategize (2016)

You should therefore aim to find out quickly if anything is wrong if your strategy, and if you need to fail, then fail fast. While a late pivot can happen, you should avoid it because the later it occurs, the more difficult and costly it is likely to be (Pichler, R., Strategize, 2016).

By building, measuring, and learning, designers are able to get closer to great user experiences sooner rather than later (Gothelf, J., & Seiden, J., Lean UX: Applying lean principles to improve user experience, 2013).

Avoid These Common Anti-patterns

As you design and run your assumption tests, keep these common anti-patterns in mind (Torres, T., Continuous Discovery Habits, 2021):

• Overly complex simulations. Some teams spend countless hours, days, or weeks trying to design and develop the perfect simulation. It’s easy to lose sight of the goal. In your first round of testing, you are looking to design fast tests that will help you gather quick signals. Design your tests to be completed in a day or two or a week at most. This will ensure that you can keep your discovery iterations high.
• Using percentages instead of specific numbers when defining evaluation criteria. Many teams equate 70% and 7 out of 10. So instead of defining their evaluation criteria as 7 out of 10, they tend to favor the percentage. These sound equivalent, but they aren’t. First, when testing with small numbers, we can’t conclude that 7 out of 10 will continue to mean 70% as our participant size grows. We want to make sure that we don’t draw too strong a conclusion from our small signals. Second, and more importantly, “70%” is ambiguous. If we test with 10 people and only 6 exhibit our desired behavior, some might conclude that the test failed. Others might argue that we need to test with more people. Be explicit from the get-go about how many people you will test with when defining your success criteria.
• Not defining enough evaluation criteria. It’s easy to forget important evaluation criteria. At a minimum, you need to define how many people to test with and how many will exhibit the desired behavior. But for some tests, defining the desired behavior may involve more than one number. For example, if your test involves sending an email, you might need to define how many people will receive the email, how long you’ll give them to open the email, and whether your success criteria is “opens” or “clicks.” Pay particular attention to the success threshold. Complex actions may require multiple measurements (e.g., opens the email, clicks on the link, takes an action).
• Testing with the wrong audience. Make sure that you are testing with the right people. If you are testing solutions for a specific target opportunity, make sure that your participants experience the need, pain point, or desire represented by that target opportunity. Remember to recruit for variation. Don’t just test with the easiest audience to reach or the most vocal audience.
• Designing for less than the best-case scenario. When testing with small numbers, design your assumption tests such that they are likely to pass. If your assumption test passes with the most likely audience, you can expand your reach to tougher audiences. This might feel like cheating, but you’ll be surprised how often your assumption tests fail. If you fail in the best-case scenario, your results will be less ambiguous. If your test fails with a less-than-ideal audience, someone on the team is going to argue you tested with the wrong audience, and you’ll have to run the test again. Remember, we want to design our tests for learning as much as we can from failures.

The Right Time for Testing Business Ideas

“You should ask yourself the question “Do people want my product?” all the time—right when you have an idea, when you make a lot of progress with building and developing the product, and definitely after you launch it. Keep doing that. By asking the question before you build the product, feature, or service, you are reducing the waste — of time, resources, and energy. The more you learn about what people want before you build anything, the less time and effort you will spend on redundant code, hundreds of hours of irrelevant meetings, and the negative emotions of team members when they realize they wasted their blood, sweat, and tears on something nobody wanted (Sharon, T., Validating Product Ideas, 2016).

You might be asking yourself, “These are all great, but when should I be doing what?”. Without knowing what kind of team setup you have and what kinds of processes you run in your organization, the best I can do is to map all of the techniques above the Double Diamond framework.

The Double Diamond Framework

Design Council’s Double Diamond clearly conveys a design process to designers and non-designers alike. The two diamonds represent a process of exploring an issue more widely or deeply (divergent thinking) and then taking focused action (convergent thinking).  

• Discover. The first diamond helps people understand, rather than assume, what the problem is. It involves speaking to and spending time with people who are affected by the issues.
• Define. The insights gathered from the discovery phase can help you to define the challenge differently.
• Develop. The second diamond encourages people to give different answers to the clearly defined problem, seeking inspiration from elsewhere and co-designing with a range of different people.
• Deliver. Delivery involves testing out different solutions at a small scale, rejecting those that will not work, and improving the ones that will.

Map of Testing Business Ideas Activities and Methods

Process Awareness characterizes the degree to which the participants are informed about the process procedures, rules, requirements, workflow, and other details. The higher the process awareness, the more profoundly the participants are engaged in a process, and so the better results they deliver.

In my experience, the biggest disconnect between the work designers need to do, and the mindset of every other team member in a team is usually about how quickly we tend — when not facilitated — to jump to solutions instead of contemplating and exploring the problem space a little longer.

Knowing when teams should be diverging, when they should be exploring, and when they should closing will help ensure they get the best out of their collective brainstorming and multiple perspectives’ power and keep the team engaged.

Testing Business Ideas during “Discover”

This phase has the highest level of ambiguity, so creating shared understanding by having a strong shared vision and good problem framing is critical. Testing Business Ideas in this phase is probably the best way to increase your level of confidence that you’ve got the right problem framing.

Here are my recommendations for suggested quantifying and qualifying activities and methods:

• User Research
• Hypothesis Writing
• Problem Framing
• Challenge Briefs
• Visioneering
• Value Proposition Design
• Jobs to be Done (JTBD)
• Testing Business Ideas
• A Value Opportunity Analysis (VOA)
• Desirability Testing

Testing Business Ideas during “Define”

In this phase, we should see the level of ambiguity diminishing, and facilitating investment discussions have the highest payoff in mitigating back-and-forth. That said, the cost of changing your mind increases drastically in this phase. Helping the team with creating great choices is critical, and experimentation should provide the data that will help them make good decisions.

Here are my recommendations for suggested quantifying and qualifying activities and methods:

• User Story Mapping
• Stories/Epics
• Design Sprints / Studio
• Concept Validation
• Outcome-Driven Innovation / JTBD
• Importance vs. Satisfaction Framework
• Kano Model
• Objectives, Goals, Strategy & Measures (OGSM)
• Product Backlog & Sprint Planning

Testing Business Ideas during “Develop”

In this phase, we should be starting to capture signals to decide if we should persevere, pivot, or stop. Since this is the phase we are moving away from simulations, and we can put something that resembles the final product in front of customers and users, we should be focusing as much as possible on capturing both preference and performance data from concept validation and usability testing.

Here are my recommendations for suggested quantifying and qualifying activities and methods:

• User Story Mapping
• Design Studio
• Specifications
• Collaborative Prototyping
• UXI Matrix (Pugh Matrix)
• Usability Testing
• Usefulness, Satisfaction, and Ease of Use (USE)
• American Customer Satisfaction Index (ACSI)
• System Usability Scale (SUS)
• Usability Metric for User Experience (UMUX)
• UMUX-Lite

Testing Business Ideas during “Deliver”

In this phase, it is probably too late to be testing business ideas, so the visibility and traceability systems should be collecting data from real customer usage, and helping us make hard choices about pivot, persevere, or stop on the next iteration of the product.

On the other hand — since the product is now in the hand of customers and users — we should be able to collect the richest data from live usage that can inform decisions about our viability hypothesis, enabling you to adjust strategic choices accordingly.

Here are my recommendations for suggested quantifying and qualifying activities and methods:

• Designer – Developer Pairing
• Fit-and-Finish
• Pirate Metrics (a.k.a. AARRR!)
• UXI Matrix (Pugh Matrix)
• Objectives, Goals, Strategy & Measures (OGSM)

Facilitating discussions around Testing Business Ideas

I’m of the opinion that designers — instead of complaining that everyone else is jumping too quickly into solutions — should facilitate the discussions and help others raise awareness around the creative and problem-solving process.

I argue for the Need for Facilitation in the sense that — if designers want to influence the decisions that shape strategy — we must step up to the plate and become skilled facilitators that respond, prod, encourage, guide, coach, and teach as they guide individuals and groups to make decisions that are critical in the business world through effective processes.

That said, my opinion is that facilitation here does not only means “facilitate workshops”, but to facilitate the decisions regardless of what kinds of activities are required.

Recommended Reading

Bland, D. J., & Osterwalder, A. (2020). Testing business ideas: A field guide for rapid experimentation. Standards Information Network.

Bland, D. J. (2020). How to select the next best test from the experiment library. Retrieved July 25, 2022, from Strategyzer.com website: https://www.strategyzer.com/blog/how-to-select-the-next-best-test-from-the-experiment-library

Brown, T., & Katz, B. (2009). Change by design: how design thinking transforms organizations and inspires innovation. [New York]: Harper Business

Croll, A., & Yoskovitz, B. (2013). Lean Analytics: Use Data to Build a Better Startup Faster. O’Reilly Media.

Design Council. (2015, March 17). What is the framework for innovation? Design Council’s evolved Double Diamond. Retrieved August 5, 2021, from designcouncil.ork.uk website: https://www.designcouncil.org.uk/news-opinion/what-framework-innovation-design-councils-evolved-double-diamond

Gadvi, V., (2022), How to identify your North Star Metric, retrieved 22 September 2022 from Mind the Product website https://www.mindtheproduct.com/how-to-identify-your-north-star-metric/

Garbugli, É. (2020). Solving Product: Reveal Gaps, Ignite Growth, and Accelerate Any Tech Product with Customer Research. Wroclaw, Poland: Amazon.

Gothelf, J. (2019, November 8). The hypothesis prioritization canvas. Retrieved April 25, 2021, from Jeffgothelf.com website: https://jeffgothelf.com/blog/the-hypothesis-prioritization-canvas/

Gothelf, J., & Seiden, J. (2021). Lean UX: Applying lean principles to improve user experience. Sebastopol, CA: O’Reilly Media.

Gothelf, J., & Seiden, J. (2017). Sense and respond: How successful organizations listen to customers and create new products continuously. Boston, MA: Harvard Business Review Press.

Govindarajan, V., & Trimble, C. (2010). The other side of innovation: Solving the execution challenge. Boston, MA: Harvard Business Review Press.

Griffiths, C., & Costi, M. (2019). The Creative Thinking Handbook: Your step-by-step guide to problem solving in business. London, England: Kogan Page.

Hanington, B., & Martin, B. (2012). Universal methods of design: 100 Ways to research complex problems, develop innovative ideas, and design effective solutions. Beverly, MA: Rockport.

Kohavi, R., & Thomke, S. (2017). The Surprising Power of Online Experiments. Harvard Business Review, (September-October 2017).

Kourdi, J. (2015). Business Strategy: A guide to effective decision-making. New York, NY: PublicAffairs

Lafley, A.G., Martin, R. L., (2013), Playing to Win: How Strategy Really Works, Publisher: Harvard Business Review Press (5 Feb 2013)

Lockwood, T., “Design Value: A Framework for Measurement” in Building Design Strategy: Using Design to Achieve Key Business Objectives, Lockwood, T., Walton, T., (2008); Allworth Press; 1 edition (November 11, 2008)

Luca, M., & Bazerman, M. H. (2021). The power of experiments: Decision making in a data-driven world. London, England: MIT Press.

Moran, K. (2016, February 28). Using the Microsoft desirability toolkit to test visual appeal. Retrieved September 9, 2022, from Nielsen Norman Group website: https://www.nngroup.com/articles/microsoft-desirability-toolkit/

Mueller, S., & Dhar, J. (2019). The decision maker’s playbook: 12 Mental tactics for thinking more clearly, navigating uncertainty, and making smarter choices. Harlow, England: FT Publishing International.

Olsen, D. (2015). The lean product playbook: How to innovate with minimum viable products and rapid customer feedback (1st ed.). Nashville, TN: John Wiley & Sons.

Osterwalder, A. (2017). How Strong is Your Innovation Evidence? Retrieved December 24, 2021, from Strategyzer.com website: https://www.strategyzer.com/blog/how-strong-is-your-innovation-evidence

Ostrovsky, M., & Schwarz, M. (2011). Reserve prices in internet advertising auctions: A field experiment. Proceedings of the 12th ACM Conference on Electronic Commerce – EC ’11. New York, New York, USA: ACM Press.

Pichler, R. (2016). Strategize: Product strategy and product roadmap practices for the digital age. Pichler Consulting.

Podeswa, H. (2008). The Business Analyst’s Handbook. Florence, AL: Delmar Cengage Learning.

Rodden, K., Hutchinson, H., & Fu, X. (2010). Measuring the user experience on a large scale: User-centered metrics for web applications. Proceedings of the 28th International Conference on Human Factors in Computing Systems – CHI ’10. New York, New York, USA: ACM Press.

Rubin, J., & Chisnell, D. (2011). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Chichester, England: John Wiley & Sons.

Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd Edition). Oxford, England: Morgan Kaufmann.

Sharon, T. (2016). Validating Product Ideas (1st Edition). Brooklyn, New York: Rosenfeld Media.

Torres, T. (2021). Continuous Discovery Habits: Discover Products that Create Customer Value and Business Value. Product Talk LLC.

Tullis, T., & Albert, W. (2013). Measuring the user experience: Collecting, analyzing, and presenting usability metrics (2nd edition). Morgan Kaufmann.

Ulwick, A. (2005). What customers want: Using outcome-driven innovation to create breakthrough products and services. Montigny-le-Bretonneux, France: McGraw-Hill.

Van Der Pijl, P., Lokitz, J., & Solomon, L. K. (2016). Design a better business: New tools, skills, and mindset for strategy and innovation. Nashville, TN: John Wiley & Sons.

Wong, R. (2021). Lean business scorecard: Desirability. Retrieved February 25, 2022, from Medium website: https://robinow.medium.com/lean-business-scorecard-desirability-ede59c82da78

Wong, R. (2021). Lean business scorecard: Feasibility. Retrieved February 25, 2022, from Medium website: https://robinow.medium.com/lean-business-scorecard-feasibility-aa36810ae779

Wong, R. (2021). Lean business scorecard: Viability. Retrieved February 25, 2022, from Medium website: https://robinow.medium.com/lean-business-scorecard-viability-de989a59aa74