The benefits of comparative testing

The benefits of comparative testing

We often get asked why we favour comparative (or binary A/B) testing over presenting survey stimuli individually or together as a set to be ranked. The truth is that in designing research there’s often no one “right way”, and each project may require something a bit different. But in terms of designing a process that’s quick and easy for both test makers and test takers, simple comparative testing is often hard to beat.

‍

For test takers:

• It fosters detailed comparative evaluation. When respondents see both options simultaneously, they can make more informed comparisons, considering the relative strengths and weaknesses of each design. It can also be more engaging for respondents to compare two options, leading to more thoughtful and considered feedback.
• Ranking multiple options simultaneously can be overwhelming for respondents. Limiting comparisons to just binary options at a time reduces cognitive load and often allows for more precise and considered feedback.
• Binary comparative testing can help people to express preferences more clearly. Some people find it easier to articulate preferences and the reasons behind them when they’re being encouraged to make a simple choice, and forcing a contrast can help to reduce respondent uncertainty by providing direct points of reference.
• Asking people to compare two options can enhance respondent focus. It can mean that respondents are encouraged to attend to specific design elements that differ or are shared between stimuli, leading to richer feedback.

For test makers:

• You often get richer data on preferences back. With Stickybeak, you find out not just which option is preferred, but also why - particularly when combined with our followup questions on the “winning” option. Comparison testing can also highlight which aspects of a design are most important to audiences.
• You frequently get more robust results. Comparative feedback can reduce the variability and potential bias that might come from evaluating designs in isolation, leading to more consistent and reliable data. Comparative contexts also more closely represent real world choice situations, unlike the isolated reactions some methodologies lean on.
• You typically get more actionable results for decision-making. It’s often easier to identify a “winner” between various concepts while consumer priorities and preferences are often easier to identify.
• In contrast to ranked lists, the pairwise preference results that are generated by binary comparisons often surface insights that aren’t obtainable by a simple ranking process.

‍

See how easy it is to create your test!

‍

‍

We’ve built Stickybeak to be easy to use while providing robust insights. If you’d like to ask about a potential project, or need a hand with designing your test, get in touch, we love to talk research! 

David Talbot is one of New Zealand's best known researchers and is an advisor to the New Zealand Labour Party.