Tag Archives: a/b testing

Incoherence: The greatest failure in most testing programs and how to avoid it

Mind reading is not part of a marketer’s job description, but mind mapping should be.

Why? Because building a mental roadmap of the thinking process during the buyer’s journey is crucial to achieving maximum conversion, and this can only be accomplished through a rigorous testing program. There are no shortcuts.

The articulation of this roadmap must be simple and precise. We call this coherence. Flint McGlaughlin explains more:

 

If you haven’t heard us say it before, we will say it again: A good idea in a brainstorming session is insufficient. You must generate more than one hypothesis from an idea and then determine which one/s should be tested. This intersection of science and art helps you achieve coherence in your understanding of customer behavior. Watch now:

 

Lastly, Flint gives a tip on avoiding incoherence in your testing efforts: Why using “and” in your hypothesis can weaken its effectiveness.


If you’d like to get better business results and learn more about MECLABS methodology that has helped capture more than $500 million in test wins, visit our Research Services page.

The post Incoherence: The greatest failure in most testing programs and how to avoid it appeared first on MarketingExperiments.

Conversion Optimization: Eight considerations to take into account when A/B testing in mobile

I’m writing this article on a laptop computer at my desk. And in your marketing department or agency, you likely do most of your work on a computer as well.

This can cause a serious disconnect with your customers as you design A/B tests.

Because more than half (52.4% according to Statista) of global internet traffic comes from a mobile device.

So, I interviewed Rebecca Strally, Director of Optimization and Design, and Todd Barrow, Director of Application Development, for tips on what considerations you should make for mobile devices when you’re planning and rolling out your tests. Rebecca and Todd are my colleagues here at MECLABS Institute (parent research organization of MarketingExperiments).

Consideration #1: Amount of mobile traffic and conversions

Just because half of global traffic is from mobile devices doesn’t mean half of your site’s traffic is from mobile devices. It could be considerably less. Or more.

Not to mention, traffic is far from the only consideration. “You might get only 30% of traffic from mobile but 60% of conversions, for example. Don’t just look at traffic. Understand the true impact of mobile on your KPIs,” Rebecca said.

Consideration #2: Mobile first when designing responsive

Even if mobile is a minority of your traffic and/or conversions, Rebecca recommends you think mobile first. For two reasons.

First, many companies measure KPIs (key performance indicators) in the aggregate, so underperformance on mobile could torpedo your whole test if you’re not careful. Not because the hypothesis didn’t work, but because you didn’t translate it well for mobile.

Second, it’s easier to go from simpler to more complex with your treatments. And mobile’s smaller form factor necessitates simplicity.

“Desktop is wide and shallow. Mobile is tall and thin. For some treatments, that can really affect how value is communicated.”  — Rebecca Strally

“Desktop is wide and shallow. Mobile is tall and thin. For some treatments, that can really affect how value is communicated,” she said.

Rebecca gave an example of a test that was planned on desktop first for a travel website. There were three boxes with value claims, and a wizard below it. On desktop, visitors could quickly see and use the wizard. The boxes offered supporting value.

But on mobile, the responsive design stacked the boxes shifting the wizard far down the page. “We had to go back to the drawing board. We didn’t have to change the hypothesis, but we had to change how it was executed on mobile,” Rebecca said.

Consideration #3: Unique impacts of mobile on what you’re testing

A smartphone isn’t just a smaller computer. It’s an entirely different device that offers different functionality. So, it’s important to consider how that functionality might affect conversions and to keep mobile-specific functionality in mind when designing tests that will be experienced by customers on both platforms — desktop and mobile.

Some examples include:

  • With the prevalence of digital wallets like Apple Pay and Google Pay, forms and credit card info is more likely to prefill. This could reduce friction in a mobile experience, and make the checkout process quicker. So while some experiences might require more value on desktop to help keep the customer’s momentum moving through the checkout process, including that value on mobile could actually slow down an otherwise friction-lite experience.
  • To speed load time and save data, customers are more likely to use ad blockers that can block popups and hosted forms. If those popups and forms contain critical information, visitors may assume your site is having a problem and not realize they are blocking this information. This may require clearly providing text explaining about the form or providing an alternative way to get the information, a step that may not be necessary on desktop.
  • Customers are touching and swiping, not typing and clicking. So information and navigation requests need to be kept simpler and lighter than on desktop.
  • Visitors can click to call. You may want to test making a phone call a more prominent call to action in mobile, while on desktop that same CTA may induce too much friction and anxiety.
  • Location services are more commonly used, providing the opportunity to better tap into customer motivation by customizing offers and information in real time and more prominently featuring brick-and-mortar related calls to action, as opposed to desktop, which is in a static location, and the user may be interested in obtaining more information before acting (which may require leaving their current location).
  • Users are accustomed to app-based experiences, so the functionality of the landing page may be more important on mobile than it is on desktop.

Consideration #4: The device may not be the only thing that’s different

“Is mobile a segment or device?” Rebecca pondered in my office.

She expanded on that thought, “Do we treat mobile like it is the same audience with the same motivations, expected actions, etc., but just on a different physical device? Or should we be treating those on mobile like a completely different segment/audience of traffic because their motivations, expected actions, etc., are different?”

She gave an example of working with a company her team was performing research services for. On this company’s website, younger people were visiting on mobile while older people were visiting on desktop. “It’s wasn’t just about a phone, it was a different collection of human beings,” she said.

Consideration #5: QA to avoid validity threats

When you’re engaged in conversion optimization testing, don’t overlook the need for quality assurance (QA) testing. If a treatment doesn’t render correctly on a mobile device, it could be that the technical difficulty is causing the change in results, not the changes you made to the treatment. If you are unaware of this, it will mislead you about the effectiveness of your changes.

This is a validity threat known as instrumentation effect.

Here are some of the devices our developers use for QAing.

(side note: That isn’t a stock photo. It’s an actual picture by Senior Designer James White. When I said it looked too much like a stock image, Associate Director of Design Lauren Leonard suggested I let the readers know “we let the designers get involved, and they got super excited about it.”)

 

“If your audience are heavy users of Safari on iPhone, then check on the actual device. Don’t rely on an emulator.”   — Todd Barrow

“Know your audience. If your audience are heavy users of Safari on iPhone, then check on the actual device. Don’t rely on an emulator. It’s rare, but depending on what you’re doing, there are things that won’t show up as a problem in an emulator. Understand what your traffic uses and QA your mobile landing pages on the actual physical devices for the top 80%,” Todd advised.

Consideration #6: The customer’s mindset

Customers may go to the same exact landing page with a very different intent when they’re coming from mobile. For example, Rebecca recounted an experiment with an auto repair chain. For store location pages, desktop visitors tended to look for coupons or more info on services. But mobile visitors just wanted to make a quick call.

“Where is the customer in the thought sequence? Mobile can do better with instant gratification campaigns related to brick-and-mortar products and services,” she said.

Consideration #7: Screen sizes and devices are not the same things

Most analytics platforms give you an opportunity to monitor your metrics based on device types, like desktop, mobile and tablet. They likely also give you the opportunity to get metrics on screen resolutions (like 1366×768 or 1920×1080).

Just keep in mind, people aren’t always viewing your websites at the size of their screen. You only know the size of the monitor not the size of the browser.

“The user could be recorded as a full-size desktop resolution, but only be viewing in a shrunken window, which may be shrunk down enough to see the tablet experience or even phone experience,” Todd said. “Bottom line is you can’t assume the screen resolutions reported in the analytics platform is actually what they were viewing the page at.”

Consideration #8: Make sure your tracking is set up correctly

Mobile can present a few unique challenges for tracking your results through your analytics and testing platforms. So make sure your tracking is set up correctly before you launch the test.

For example, if you’re using a tag control manager and tagging things through it based on CSS properties, if the page shifts at different breakpoints that change the page structure, you could have an issue.

“If you’re tagging a button based on its page location at the bottom right, but then it gets relocated on mobile, make sure you’re accounting for that,” Todd advised.

Also, understand how the data is being communicated. “Because Google Tag Manager and Google Optimize are asynchronous, you can get mismatched data if you don’t follow the best practices,” Todd said.

“If you see in your data that the control has twice as many hits as the treatment, there is a high probability you’ve implemented something in a way that didn’t account for the way asynchronous tags work.”                  —Todd Barrow

Todd provided a hard-coded page view as an example. “Something to watch for when doing redirect testing … a tracking pixel could fire before the page loads and does the split. If you see in your data that the control has twice as many hits as the treatment, there is a high probability you’ve implemented something in a way that didn’t account for the way asynchronous tags work. This is really common,” Todd said.

“If you know that’s going to happen, you can segment the data to clean it,” he said.

Related Resources

Free Mobile Conversion Micro Class from MECLABS Institute

Mobile Marketing: What a 34% increase in conversion rate can teach you about optimizing for video

Mobile Marketing: Optimizing the evolving landscape of mobile email marketing

Mobile Conversion Optimization Research Services: Get better mobile results from deeper customer understanding

The post Conversion Optimization: Eight considerations to take into account when A/B testing in mobile appeared first on MarketingExperiments.

Mobile A/B Testing: Quality assurance checklist

Real-world behavioral tests are an effective way to better understand your customers and optimize your conversion rates. But for this testing to be effective, you must make sure it is accurately measuring customer behavior.

One reason these A/B split tests fail to give a correct representation of customer behavior is because of validity threats. This series of checklists is designed to help you overcome Instrumentation Effect. It is based on actual processes used by MECLABS Institute’s designers, developers and analysts when conducting our research services to help companies improve marketing performance.

MECLABS defines Instrumentation Effect as “the effect on the test variable caused by a variable external to an experiment, which is associated with a change in the measurement instrument.” In other words, the results you see do not come from the change you made (say, a different headline or layout), but rather, because some of your technology has affected the results (slowed load time, miscounted analytics, etc.)

Avoiding Instrumentation Effect is even more challenging for any test that will have traffic from mobile devices (which today is almost every test). So, to help you avoid the Instrumentation Effect validity threat, we’re providing the following QA checklist. This is not meant for you to follow verbatim, but to serve as a good jumping-off point to make sure your mobile tests are technically sound. For example, other browsers than the ones listed here may be more important for your site’s mobile functionality. Maybe your landing page doesn’t have a form, or you may use different testing tools, etc.

Of course, effective mobile tests require much more than thorough QA — you also must know what to test to improve results. If you’re looking for ideas for your tests that include mobile traffic, you can register for the free Mobile Conversion micro course from MECLABS Institute based on 25 years of conversion optimization research (with increasing emphases on mobile traffic in the last half decade or so).

There’s a lot of information here, and different people will want to save this checklist in different ways. You can scroll through the article you’re on to see the key steps of the checklist. Or use the form on this page to download a PDF of the checklist.

 

Roles Defined

The following checklists are broken out by teams serving specific roles in the overall mobile development and A/B testing process. The checklists are designed to help cross-functional teams, with the benefit being that multiple people in multiple roles bring their own viewpoint and expertise to the project and evaluate whether the mobile landing page and A/B testing are functioning properly before launch and once it is live.

For this reason, if you have people serving multiple roles (or you’re a solopreneur and do all the work yourself), these checklists may be repetitive for you.

Here is a quick look at each team’s overall function in the mobile landing page testing process, along with the unique value it brings to QA:

Dev Team These are the people who build your software and websites, which could include both front-end development and back-end development. They use web development skills to create websites, landing pages and web applications.

For many companies, quality assurance (QA) would fall in this department as well, with the QA team completing technical and web testing. While a technical QA person is an important member of the team for ensuring you run valid mobile tests, we have included other functional areas in this QA checklist because different viewpoints from different departments will help decrease the likelihood of error. Each department has its own unique expertise and is more likely to notice specific types of errors.

Value in QA: The developers and technological people are most likely to notice any errors in the code or scripts and make sure that the code is compatible with all necessary devices.

 

Project Team – Depending on the size of the organization, this may be a dedicated project management team, a single IT or business project manager, or a passionate marketing manager keeping track of and pushing to get everything done.

It is the person or team in your organization that coordinates work and manages timelines across multiple teams, ensures project work is progressing as planned and that project objectives are being met.

Value in QA: In addition to making sure the QA doesn’t take the project off track and threaten the launch dates of the mobile landing page test, the project team are the people most likely to notice when business requirements are not being met.

 

Data Team The data scientist(s), analyst(s) or statistician(s) helped establish the measure of success (KPI – key performance indicator) and will monitor the results for the test. They will segment and gather the data in the analytics platform and assemble the report explaining the test results after they have been analyzed and interpreted.

Value in QA: They are the people most likely to notice any tracking issues from the mobile landing page not reporting events and results correctly to the analytics platform.

 

Design Team The data scientist(s), analyst(s) or statistician(s) helped establish the measure of success (KPI – key performance indicator) and will monitor the results for the test. They will segment and gather the data in the analytics platform and assemble the report explaining the test results after they have been analyzed and interpreted.

Value in QA: They are the people most likely to notice any tracking issues from the mobile landing page not reporting events and results correctly to the analytics platform.

 

DEV QA CHECKLIST

Pre-launch, both initial QA and on Production where applicable

Visual Inspection and Conformity to Design of Page Details

  • Verify latest copy in place
  • Preliminary checks in a “reference browser” to verify design matches latest comp for desktop/tablet/mobile layouts
  • Use the Pixel Perfect Overlay function in Firefox Developer Tools – The purpose of this tool is to take an image that was provided by the designer and lay it over the website that was produced by the developer. The image is a transparency which you can use to point out any differences or missing elements between the design images and the webpage.
  • Displaying of images – Make sure that all images are displaying, aligned and up to spec with the design.
  • Forms, List and Input Elements (Radio Buttons, Click Boxes) – Radio buttons (Dots and Circles) and Checkboxes (Checks and Boxes) are to be tested thoroughly as they may trigger secondary actions. For example, selecting a “Pay by Mail” radio button will sometimes automatically hide the credit card form.
  • Margins and Borders – Many times, you will notice that a portion of the body or perhaps a customer review or image is surrounded by a border or maybe even the whole page. It is our duty to inspect them so that there are no breaks and that they’re prominent enough for the user to decipher each bordered section.
  • Copy accuracy – Consistency between typography, capitalization, punctuation, quotations, hyphens, dashes, etc. The copy noted in the webpage should match any documents provided pertaining to copy and text unless otherwise noted or verified by the project manager/project sponsor.
  • Font styling (Font Color, Format, Style and Size) – To ensure consistency with design, make sure to apply the basic rules of hierarchy for headers across different text modules such as titles, headers, body paragraphs and legal copies.
  • Link(s) (Color, Underline, Clickable)

Web Page Functionality: Verify all page functionality works as expected (ensure treatment changes didn’t impact page functionality)

  • Top navigation functionality – Top menu, side menu, breadcrumb, anchor(s)
  • Links and redirects are correct
  • Media – Video, images, slideshow, PDF, audio
  • Form input elements – drop down, text fields, check and radio module, fancy/modal box
  • Form validation – Error notification, client-side errors, server-side errors, action upon form completion (submission confirmation), SQL injection
  • Full Page Functionality – Search function, load time, JavaScript errors
  • W3C Validation – CSS Validator (http://jigsaw.w3.org/css-validator/), markup validator (http://validator.w3.org/)
  • Verify split functional per targeting requirements
  • Verify key conversion scenario (e.g., complete a test order, send test email from email system, etc.) – If not already clear, QA lead should verify with project team how test orders should be placed
  • Where possible, visit the page as a user would to ensure targeting parameters are working properly (e.g., use URL from the PPC ad or email, search result, etc.)

Tracking Metrics

  • Verify tracking metrics are firing in browser, and metric names match requirements – Check de-bugger to see firing as expected
  • Verify reporting within the test/analytics tool where possible – Success metrics and click tracking in Adobe Target, Google Content Experiments, Google Analytics, Optimizely, Floodlight analytics, email data collection, etc.

Back End Admin Panel

Notify Project Team and Data Team it is ready for their QA (via email preferably) – indicate what reference browser is. After Project Team initial review, complete full cross browser/ cross device checks using “reference browser” as a guide:

Browser Functionality – Windows

  • Internet Explorer 7 (IE7)
  • IE8
  • IE9
  • IE10
  • IE11
  • Modern Firefox
  • Modern Chrome

Browser Functionality – macOS

  • Modern Safari
  • Modern Chrome
  • Modern Firefox8

Mobile Functionality – Tablet

  • Android
  • Windows
  • iOS

Mobile Functionality – Mobile phone

  • Android
  • Windows
  • iOS

Post-launch, after the test is live to the public:

  • Notify Project Team & Data Team the test is live and ready for post-launch review (via email preferably)
  • Verify split is open to public Verify split functional per targeting requirements
  • Where possible, visit the page as a user would to ensure targeting parameters are working properly (e.g., use URL from the PPC ad or email, search result, etc.)
  • Test invalid credit cards on a production environment
PROJECT TEAM QA CHECKLIST:

Pre-Launch and Post-Launch QA:

  • Check that copy and design are correct for control and treatments in the “reference browser”:
  • Ensure all added copy/design elements are there and correct
  • Ensure all removed copy/design elements are gone
  • Ensure all changed copy/design elements are correct
  • Ensure control experience is as intended for the test
  • Check page functionality:
  • Ensure all added/changed functionality is working as expected
  • Ensure all standard/business as usual – BAU_ functionality is working as expected:
  • Go through the typical visitor path (even beyond the testing page/ location) and ensure everything functions as expected
  • Make sure links go where supposed to, fields work as expected, data passes as expected from page to page.
  • Check across multiple browser sizes (desktop, tablet, mobile)
  • If site is responsive, scale the browser from full screen down to mobile and check to ensure all the page breaks look correct
  • Where possible, visit the page the way a typical visitor would hit the page (e.g., through PPC Ad, organic search result, specific link/button on site, through email)
DATA QA CHECKLIST:

Pre-Launch QA Checklist (complete on Staging and Production as applicable):

  • Verify all metrics listed in the experiment design are present in analytics portal
  • Verify all new tracking metrics’ names match metrics’ names from tracking document
  • Verify all metrics are present in control and treatment(s) (where applicable)
  • Verify conversion(s) are present in control and treatment(s) (where possible)
  • Verify any metrics tracked in a secondary analytics portal (where applicable)
  • Immediately communicate any issues that arise to the dev lead and project team
  • Notify dev lead and project team when Data QA is complete (e-mail preferably)

Post-Launch QA / First Data Pull:

  • Ensure all metrics for control and treatment(s) are receiving traffic
  • Ensure traffic levels are in line with the pre-test levels used for test duration estimation
  • Update Test Duration Estimation if necessary
  • Immediately communicate any issues that arise to the project team
  • Notify dev lead and project team when first data pull is complete (e-mail preferably)
DESIGN QA CHECKLIST:

Pre-Launch Review:

  • Verify intended desktop functionality (if applicable)
  • Accordions
  • Error states
  • Fixed Elements (nav, growler, etc.)
  • Form fields
  • Hover states – desktop only
  • Links
  • Modals
  • Sliders
  • Verify intended tablet functionality (if applicable)
  • Accordions
  • Error states
  • Fixed Elements (nav, growler, etc.)
  • Form fields
  • Gestures – touch device only
  • Links
  • Modals
  • Responsive navigation
  • Sliders
  • Verify intended mobile functionality (if applicable)
  • Accordions
  • Error states
  • Fixed Elements (nav, growler, etc.)
  • Form fields
  • Gestures – touch device only
  • Links
  • Modals
  • Responsive navigation
  • Sliders
  • Verify layout, spacing and flow of elements
  • Padding/Margin
  • “In-between” breakpoint layouts (as these are not visible in the comps)
  • Any “of note” screen sizes that may affect test goals (For example: small laptop 1366×768 pixels, 620px of height visibility)
  • Verify imagery accuracy, sizing and placement
  • Images (Usually slices Design provided to Dev)
  • Icons (Could be image, svg or font)
  • Verify Typography
  • Color
  • Font-size
  • Font-weight
  • Font-family
  • Line-height

Qualifying questions, if discrepancies are found:

  • Is there an extremely strict adherence to brand standards?
  • Does it impact the hierarchy of the page information?
  • Does it appear broken/less credible?
  • Immediately communicate any issues that arise to the dev lead and project team
  • Notify dev lead and project team when data QA is complete (e-mail preferably)

To download a free PDF of this checklist, simply complete the below form.


___________________________________________________________________________________

Increase Your Mobile Conversion Rates: New micro course 

Hopefully, this Mobile QA Checklist helps your team successfully launch tests that have mobile traffic. But you still may be left with the question — what should I test to increase conversion?

MECLABS Institute has created five micro classes (each under 12 minutes) based on 25 years of research to help you maximize the impact of your messages in a mobile environment.

In the complimentary Mobile Conversion Micro Course, you will learn:

  • The 4 most important elements to consider when optimizing mobile messaging
  • How a large telecom company increased subscriptions in a mobile cart by 16%
  • How the same change in desktop and mobile environments had opposing effects on conversion

Register Now for Free

The post Mobile A/B Testing: Quality assurance checklist appeared first on MarketingExperiments.

Designing Hypotheses that Win: A four-step framework for gaining customer wisdom and generating marketing results

There are smart marketers everywhere testing many smart ideas — and bad ones. The problem with ideas is that they are unreliable and unpredictable. Knowing how to test is only half of the equation. As marketing tools and technology evolve rapidly offering new, more powerful ways to measure consumer behavior and conduct more sophisticated testing, it is becoming more important than ever to have a reliable system for deciding what to test.

Without a guiding framework, we are left to draw ideas almost arbitrarily from competitors, brainstorms, colleagues, books and any other sources without truly understanding what makes them good, bad or successful. Ideas are unpredictable because until you can articulate a forceful “because” statement to why your ideas will work, regardless of how good, they are nothing more than a guess, albeit educated, but most often not by the customer.

20+ years of in-depth research, testing, optimization and over 20,000+ sales path experiments have taught us that there is an answer to this problem, and that answer involves rethinking how we view testing and optimization. This short article touches on the keynote message MECLABS Institute’s founder Flint McGlaughlin will give at the upcoming 2018 A/B Testing Summit virtual conference on December 12-13th.  You can register for free at the link above.

Marketers don’t need better ideas; they need a better understanding of their customer.

So if understanding your customer is the key to efficient and effective optimization and ideas aren’t reliable or predictable, what then? We begin with the process of intensively analyzing existing data, metrics, reports and research to construct our best Customer Theory, which is the articulation of our understanding of our customer and their behavior toward our offer.

Then, as we identify problems/focus areas for higher performance in our funnel, we transform our ideas for solving them into a hypothesis containing four key parts:

  1. If [we achieve this in the mind of the consumer]
  2. By [adding, subtracting or changing these elements]
  3. Then [this result will occur]
  4. Because [that will confirm or deny this belief/hypothesis about the customer]

By transforming ideas into hypotheses, we orient our test to learn about our customer rather than merely trying out an idea. The hypothesis grounds our thinking in the psychology of the customer by providing a framework that forces the right questions into the equation of what to test. “The goal of a test is not to get a lift, but to get a learning,” says Flint McGlaughlin, “and learning compounds over time.”

Let’s look at some examples of what to avoid in your testing, along with good examples of hypotheses.

Not this:

“Let’s advertise our top products in our rotating banner — that’s what Competitor X is doing.”

“We need more attractive imagery … Let’s place a big, powerful hero image as our banner. Everyone is doing it.”

“We should go minimalist … It’s modern, sleek and sexy, and customers love it. It’ll be good for our brand. Less is more.”

But this:

 “If we emphasize and sample the diversity of our product line by grouping our top products from various categories in a slowly rotating banner, we will increase clickthrough and engagement from the homepage because customers want to understand the range of what we have to offer (versus some other value, e.g., quality, style, efficacy, affordability, etc.).”

“If we reinforce the clarity of the value proposition by using more relevant imagery to draw attention to the most important information, we will increase clickthrough and ultimately conversion because the customer wants to quickly understand why we’re different in such a competitive space.”

“If we better emphasize the primary message be reducing unnecessary, less-relevant page elements and changing to a simpler, clearer more readable design, we will increase clickthrough and engagement on the homepage because customers are currently overwhelmed by too much friction on this page.”

The golden rule of optimization is “Specificity converts. The more specific/relevant you can be to the individual wants and needs of your ideal customer, the more likely the probability of conversion. To be as specific and relevant as possible to a consumer, we use testing not as merely an idea-trial hoping for positive results, but as a mechanism to fill in the gaps of our understanding that existing data can’t answer. Our understanding of the customer is what powers the efficiency and efficacy of our testing.

In Summary …

Smart ideas only work sometimes, but a framework based on understanding your customer will yield more consistent, more rewarding results that only improve over time. The first key to rethinking your approach to optimization is to construct a robust customer theory articulating your best understanding of your customer. From this, you can transform your ideas into hypotheses that will begin producing invaluable insights to lay the groundwork for how you communicate with your customer.

Looking for ideas to inform your hypotheses? We have created and compiled a 60-page guide that contains 21 crafted tools and concepts, and outlines the unique methodology we have used and tested with our partners for 20+ years. You can download the guide for free here: A Model of Your Customer’s Mind

Related Resources

A/B Testing Summit free online conference – Research your seat to see Flint McGlaughlin’s keynote Design Hypotheses that Win: A 4-step framework for gaining customer wisdom and generating significant results

The Hypothesis and the Modern-Day Marketer

Customer Theory: How we learned from a previous test to drive a 40% increase in CTR

The post Designing Hypotheses that Win: A four-step framework for gaining customer wisdom and generating marketing results appeared first on MarketingExperiments.

Green Marketing: The psychological impact of an eco-conscious marketing campaign

The following research was first published in the MECLABS Quarterly Research Digest, July 2014.

Almost every industry has seen a shift toward “green technology” or “eco-friendly materials.” While this is certainly a positive step for the earth, it can rightly be questioned whether the marketing that touts this particular aspect of the business is really effective.

Marketing offices across the globe face some very real questions:

  • Does highlighting your green practices actually cause more people to buy from you?
  • Does it have any impact at all?
  • Does it, much to our shock and dismay, temper conversion?

When we find an issue like this, we are inclined to run a test rather than trust our marketing intuition.

Experiment: Does green marketing impact conversion?

The Research Partner for Test Protocol (TP) 11009 is a furniture company wanting to increase sales of its eco-friendly mattresses. Our key tracking metric was simple: purchases. Our research question was this: Which landing page would create more mattress sales, A or B?

As you can see in Figure 1.1, the pages were identical save for one key aspect: Version B included an extra section that Version A left out. In this section, we went into more detail about the green aspects of the mattress. It should be noted, however, that both pages included the “GreenGuard Gold Certification Seal,” so it is not as if Version A is devoid of the green marketing angle. Version B simply spelled it out more clearly.

Figure 1.1

Did the change make a difference? Yes, Version B outperformed Version A by 46%. Remember, this lift is in purchases, not simply clickthrough.

 

 

We have established that green marketing can be effective. But in what cases? How can we put that knowledge to good use and navigate the waters of green marketing with a repeatable methodology?

Four ways to create effective green marketing campaigns

In the test above, green marketing made a clear and significant difference. We made four observations as to why this particular green marketing strategy succeeded. You can use them as guides toward your own green marketing success.

Key Observation #1. The value was tangible. The value created by the copy was directly connected to the customer experience.

In the case of the GreenGuard Certified mattress, the value of being green was not solely based on its being eco-friendly. It also was customer-friendly. The green nature of the manufacturing process directly affected and increased the quality of the product. The copy stated that the mattress “meets the world’s most rigorous, third-party chemical emissions standards with strict low emission levels for over 360 volatile organic compounds.” Not only is it good for the earth, but it is also good for your toddler and your grandmother.

This tangible benefit to the customer experience is not always present in green marketing. In Figure 2.1, you see three examples of green marketing that fail to leverage a tangible benefit to the customer:

Figure 2.1

 

  1. When a hotel encourages you to reuse your towels to “save water,” it does nothing to improve the value of your experience with them. If anything, it may come off as an attempt to guilt the guest into reducing the hotel’s water bill.
  2. GE’s “Ecomagination” campaign is devoid of a tangible benefit to the customer. How does GE being green make my microwave better for me? The campaign doesn’t offer an answer.
  3. Conversely, “100% recycled toilet tissue” not only does not offer a tangible benefit to the customer, it also implies that the customer might not receive the same quality experience they would have with a non-green option.

For green marketing to optimally operate, you must be able to a point out a tangible benefit to the customer, in addition to the earth-friendly nature of the product.

Key Observation #2. The issue was relevant. The issue addressed by the copy dealt with a key concern already present in the mind of the prospect.

For people in the market for a new mattress, especially those with young children, sensitive skin or allergies, there are well-founded concerns regarding the chemicals and other materials that go into the production of the mattress. This concern already exists in the mind of the customer. It does not need to be raised or hyped by the marketer. Again, not all green marketing campaigns address relevant concerns.

Figure 3.1

 

  1. People are more concerned with safety, comfort and affordability when traveling. Whether the airline is green or not is not generally a concern.
  2. When choosing a sunscreen, most people don’t go in with aspirations of choosing a green option. Their top concern is sun protection, and biodegradable sunscreen doesn’t appear to meet that need as well as another option can.
  3. Again, “biodegradable” is not a common concern brought to the table by people buying pens.

All of these, while potentially noble causes, do not directly connect to a relevant problem the customer experiences. On the other hand, the GreenGuard Certified mattress immediately addressed a pressing concern held by the customer. It is “perfect for those with skin sensitivity or allergies.”

Key Observation #3. The claim was unique. The claim of exclusivity in the copy intensified the “only” factor of the product itself.

Just like any other benefit, green marketing benefits gain or lose value based on how many others can make the claim. If a web hosting platform touts itself as green or eco-friendly, the claim doesn’t hold as much force because the industry is saturated with green options (Figure 4.1). The same is true of BPA-free water bottles (Figure 4.2).

 

Figure 4.1

 

Figure 4.2

 

However, in the case of our Research Partner, not many of its competitors could make the “GreenGuard Gold Certification” claim (Figure 4.3). This added exclusivity — not to mention that Gold status implied they achieved the highest level of certification. Uniqueness drives value up, as long as the benefit in question is actually in demand.

Figure 4.3

 

Key Observation #4. The evidence was believable. The evidence provided in the copy lent instant credibility to any of the claims.

After the initial wave of green marketing techniques and practices took the industry by storm, there was a very justified backlash against those simply trying to cash in on the trend. Lawsuits were filed against marketers exaggerating their green-ness, including the likes of SC Johnson, Fiji Water, Hyundai and others. As a result, consumers became wary of green claims and must be persuaded otherwise by believable data.

In the winning design above, we did this in three ways:

  1. Verification: “100% Certified by GreenGuard Gold”
  2. Specification: “Our mattresses get reviewed quarterly to maintain this seal of approval. Last certification: January 4th, 2014.”
  3. Quantification: “Low emission levels for over 360 volatile organic compounds.”

The ability to prove that your green practices or eco-friendly products are truly as earth-friendly — and tangibly beneficial — as you claim is a crucial component in creating a green marketing angle that produces a significant increase in conversion.

How to approach your green marketing challenges

We have seen that green marketing can work. Still, this is not a recommendation to throw green marketing language into everything you put out. Green marketing is not a cure-all.

However, given the right circumstances, the right green positioning can certainly achieve lifts, and we want you to be able to capitalize on that. Therefore, we have created this checklist to help you analyze and improve your green marketing tactics.

☐  Is your green marketing tangible?

Does the nature of the green claims actually make the end product more appealing?

☐  Is your green marketing relevant?

Does the fact that your offer is green solve an important problem in the mind of the customer?

☐  Is your green marketing unique?

Can anyone else in your vertical make similar claims? If so, how do your claims stand apart?

☐  Is your green marketing believable?

Are your claims actually true? If so, how can you quantify, verify or specify your particular claims?

Of course, this checklist is only a starting point. Testing your results is the only true way to discover if your new green techniques are truly improving conversion.

Related Resources

Learn how Research Partnerships work, and how you can join MECLABS in discovering what really works in marketing

Read this MarketingExperiments Blog post to learn how to craft the right research question

Sometimes we only have intangible benefits to market. In this interview, Tim Kachuriak, Founder and Chief Innovation & Optimization Officer, Next After, explains how to get your customers to say, “heck yes”

One way to be relevant is better understand your customers is through data-driven marketing

Discover three techniques for standing out a competitive marketing, including focusing on your “only” factor

Read on for nine elements that help make your marketing claims more believable

The post Green Marketing: The psychological impact of an eco-conscious marketing campaign appeared first on MarketingExperiments.

Green Marketing: The psychological impact of an eco-conscious marketing campaign

The following research was first published in the MECLABS Quarterly Research Digest, July 2014.

Almost every industry has seen a shift toward “green technology” or “eco-friendly materials.” While this is certainly a positive step for the earth, it can rightly be questioned whether the marketing that touts this particular aspect of the business is really effective.

Marketing offices across the globe face some very real questions:

  • Does highlighting your green practices actually cause more people to buy from you?
  • Does it have any impact at all?
  • Does it, much to our shock and dismay, temper conversion?

When we find an issue like this, we are inclined to run a test rather than trust our marketing intuition.

Experiment: Does green marketing impact conversion?

The Research Partner for Test Protocol (TP) 11009 is a furniture company wanting to increase sales of its eco-friendly mattresses. Our key tracking metric was simple: purchases. Our research question was this: Which landing page would create more mattress sales, A or B?

As you can see in Figure 1.1, the pages were identical save for one key aspect: Version B included an extra section that Version A left out. In this section, we went into more detail about the green aspects of the mattress. It should be noted, however, that both pages included the “GreenGuard Gold Certification Seal,” so it is not as if Version A is devoid of the green marketing angle. Version B simply spelled it out more clearly.

Figure 1.1

Did the change make a difference? Yes, Version B outperformed Version A by 46%. Remember, this lift is in purchases, not simply clickthrough.

 

 

We have established that green marketing can be effective. But in what cases? How can we put that knowledge to good use and navigate the waters of green marketing with a repeatable methodology?

Four ways to create effective green marketing campaigns

In the test above, green marketing made a clear and significant difference. We made four observations as to why this particular green marketing strategy succeeded. You can use them as guides toward your own green marketing success.

Key Observation #1. The value was tangible. The value created by the copy was directly connected to the customer experience.

In the case of the GreenGuard Certified mattress, the value of being green was not solely based on its being eco-friendly. It also was customer-friendly. The green nature of the manufacturing process directly affected and increased the quality of the product. The copy stated that the mattress “meets the world’s most rigorous, third-party chemical emissions standards with strict low emission levels for over 360 volatile organic compounds.” Not only is it good for the earth, but it is also good for your toddler and your grandmother.

This tangible benefit to the customer experience is not always present in green marketing. In Figure 2.1, you see three examples of green marketing that fail to leverage a tangible benefit to the customer:

Figure 2.1

 

  1. When a hotel encourages you to reuse your towels to “save water,” it does nothing to improve the value of your experience with them. If anything, it may come off as an attempt to guilt the guest into reducing the hotel’s water bill.
  2. GE’s “Ecomagination” campaign is devoid of a tangible benefit to the customer. How does GE being green make my microwave better for me? The campaign doesn’t offer an answer.
  3. Conversely, “100% recycled toilet tissue” not only does not offer a tangible benefit to the customer, it also implies that the customer might not receive the same quality experience they would have with a non-green option.

For green marketing to optimally operate, you must be able to a point out a tangible benefit to the customer, in addition to the earth-friendly nature of the product.

Key Observation #2. The issue was relevant. The issue addressed by the copy dealt with a key concern already present in the mind of the prospect.

For people in the market for a new mattress, especially those with young children, sensitive skin or allergies, there are well-founded concerns regarding the chemicals and other materials that go into the production of the mattress. This concern already exists in the mind of the customer. It does not need to be raised or hyped by the marketer. Again, not all green marketing campaigns address relevant concerns.

Figure 3.1

 

  1. People are more concerned with safety, comfort and affordability when traveling. Whether the airline is green or not is not generally a concern.
  2. When choosing a sunscreen, most people don’t go in with aspirations of choosing a green option. Their top concern is sun protection, and biodegradable sunscreen doesn’t appear to meet that need as well as another option can.
  3. Again, “biodegradable” is not a common concern brought to the table by people buying pens.

All of these, while potentially noble causes, do not directly connect to a relevant problem the customer experiences. On the other hand, the GreenGuard Certified mattress immediately addressed a pressing concern held by the customer. It is “perfect for those with skin sensitivity or allergies.”

Key Observation #3. The claim was unique. The claim of exclusivity in the copy intensified the “only” factor of the product itself.

Just like any other benefit, green marketing benefits gain or lose value based on how many others can make the claim. If a web hosting platform touts itself as green or eco-friendly, the claim doesn’t hold as much force because the industry is saturated with green options (Figure 4.1). The same is true of BPA-free water bottles (Figure 4.2).

 

Figure 4.1

 

Figure 4.2

 

However, in the case of our Research Partner, not many of its competitors could make the “GreenGuard Gold Certification” claim (Figure 4.3). This added exclusivity — not to mention that Gold status implied they achieved the highest level of certification. Uniqueness drives value up, as long as the benefit in question is actually in demand.

Figure 4.3

 

Key Observation #4. The evidence was believable. The evidence provided in the copy lent instant credibility to any of the claims.

After the initial wave of green marketing techniques and practices took the industry by storm, there was a very justified backlash against those simply trying to cash in on the trend. Lawsuits were filed against marketers exaggerating their green-ness, including the likes of SC Johnson, Fiji Water, Hyundai and others. As a result, consumers became wary of green claims and must be persuaded otherwise by believable data.

In the winning design above, we did this in three ways:

  1. Verification: “100% Certified by GreenGuard Gold”
  2. Specification: “Our mattresses get reviewed quarterly to maintain this seal of approval. Last certification: January 4th, 2014.”
  3. Quantification: “Low emission levels for over 360 volatile organic compounds.”

The ability to prove that your green practices or eco-friendly products are truly as earth-friendly — and tangibly beneficial — as you claim is a crucial component in creating a green marketing angle that produces a significant increase in conversion.

How to approach your green marketing challenges

We have seen that green marketing can work. Still, this is not a recommendation to throw green marketing language into everything you put out. Green marketing is not a cure-all.

However, given the right circumstances, the right green positioning can certainly achieve lifts, and we want you to be able to capitalize on that. Therefore, we have created this checklist to help you analyze and improve your green marketing tactics.

☐  Is your green marketing tangible?

Does the nature of the green claims actually make the end product more appealing?

☐  Is your green marketing relevant?

Does the fact that your offer is green solve an important problem in the mind of the customer?

☐  Is your green marketing unique?

Can anyone else in your vertical make similar claims? If so, how do your claims stand apart?

☐  Is your green marketing believable?

Are your claims actually true? If so, how can you quantify, verify or specify your particular claims?

Of course, this checklist is only a starting point. Testing your results is the only true way to discover if your new green techniques are truly improving conversion.

Related Resources

Learn how Research Partnerships work, and how you can join MECLABS in discovering what really works in marketing

Read this MarketingExperiments Blog post to learn how to craft the right research question

Sometimes we only have intangible benefits to market. In this interview, Tim Kachuriak, Founder and Chief Innovation & Optimization Officer, Next After, explains how to get your customers to say, “heck yes”

One way to be relevant is better understand your customers is through data-driven marketing

Discover three techniques for standing out a competitive marketing, including focusing on your “only” factor

Read on for nine elements that help make your marketing claims more believable

The post Green Marketing: The psychological impact of an eco-conscious marketing campaign appeared first on MarketingExperiments.

Get Your Free Test Discovery Tool to Help Log all the Results and Discoveries from Your Company’s Marketing Tests

Come budget time, do you have an easy way to show all the results from your testing? Not just conversion lifts, but the golden intel that senior business leaders crave — key insights into customer behavior.

To help you do that, we’ve created the free MECLABS Institute Test Discovery Tool, so you can build a custom discovery library for your organization. This simple tool is an easy way of helping your company create a repository of discoveries from its behavioral testing with customers and showing business leaders all the results of your testing efforts. Just click the link below to get yours.

 

Click Here to Download Your FREE Test Discovery Tool Instantly

(no form to fill out, just click to get your instant download of this Excel-based tool)

 

In addition to enabling you to show comprehensive test results to business leaders, a custom test discovery library for your brand helps improve your overall organization’s performance. You probably have an amazing amount of institutional knowledge stuck in your cranium. From previous campaigns and tests, you have a good sense of what will work with your customers and what will not. You probably use this info to inform future tests and campaigns, measure what works and build your knowledge base even more.

But to create a truly successful organization, you have to get that wisdom out of your head and make sure everyone in your marketing department and at your agencies has access to that valuable intel. Plus, you want the ability to learn from everyone in your organization as well.

 

Click Here to Download Your FREE Test Discovery Tool Instantly

(no form to fill out, just click to get your instant download of this Excel-based tool)

 

This tool was created to help a MECLABS Research Partner keep track of all the lessons learned from its tests.

“The goal of building this summary spreadsheet was to create a functional and precise approach to document a comprehensive summary of results. The template allows marketers to form a holistic understanding of their test outcomes in an easily digestible format, which is helpful when sharing and building upon future testing strategy within your organization. The fields within the template are key components that all testing summaries should possess to clearly understand what the test was measuring and impacting, and the validity of the results,” said Delaney Dempsey, Data Scientist, MECLABS Institute.

“Basically, the combination of these fields provides a clear understanding of what worked and what did not work. Overall, the biggest takeaway for marketers is that having an effective approach to documenting your results is an important element in creation of your customer theory and impactful marketing strategies. Ultimately, past test results are the root of our testing discovery about our customers,” she explained.

 

Click Here to Download Your FREE Test Discovery Tool Instantly

(no form to fill out, just click to get your instant download of this Excel-based tool)

 

Here is a quick overview for filling out the fields in this tool (we’ve also included this info in the tool) …

Click on the image to enlarge in new window

How to use this tool to organize your company’s customer discoveries from real-world behavioral tests

For a deeper exploration of testing, and to learn where to test, what to test and how to turn basic testing data into customer wisdom, you can take the MECLABS Institute Online Testing on-demand certification course.

Test Dashboard: This provides an overview of your tests. The info automatically pulls from the information you input for each individual test on the other sheets in this Excel document. You may decide to color code each test stream (say blue for email, green for landing pages, etc.) to more easily read the dashboard. (For instructions on adding more rows to the Test Dashboard, and thus more test worksheets to the Excel tool, scroll down to the “Adding More Tests” section.)

Your Test Name Here: Create a name for each test you run. (To add more tabs to run more tests, scroll down to the “Adding More Tests” section.)

Test Stream: Group tests in a way that makes the most sense for your organization. Some examples might be the main site, microsite, landing pages, homepage, email, specific email lists, PPC ads, social media ads and so on.

Test Location: Where in your test stream did this specific test occur? For example, if the Test Stream was your main site, the Test Location may have been on product pages, a shopping page or on the homepage. If one of your testing streams is Landing Pages, the test location may have been a Facebook landing page for a specific product.

Test Tracking Number: To organize your tests, it can help to assign each test a unique tracking number. For example, every test MECLABS Institute conducts for a company has a Test Protocol Number.

Timeframe Run: Enter the dates the test ran and the number of days it ran. MECLABS recommends you run your tests for at least a week, even if it reaches a statistically significant sample size, to help reduce the chances of a validity threat known as History Effect.

Hypothesis: The reason to run a test is to prove or disprove a hypothesis.

Do you know how you can best serve your customer to improve results? What knowledge gaps do you have about your customer? What internal debates do you have about the customer? What have you debated with your agency or vendor partner? Settle those debates and fill those knowledge gaps by crafting a hypothesis and running a test to measure real-world customer behavior.

Here is the approach MECLABS uses to formulate a hypothesis, with an example filled in …

# of Treatments: This is the number of versions you are testing. For example, if you had Landing Page A and Landing Page B, that would be two treatments. The more treatments you test in one experiment, the more samples you need to avoid a Sampling Distortion Effect validity threat, which can occur when you do not collect a significant number of observations.

Valid/Not Valid: A valid test measures what it claims to measure. Valid tests are well-founded and correspond accurately to the real world. Results of a valid test can be trusted to be accurate and to represent real-world conditions. Invalid tests fail to measure what they claim to measure and cannot be trusted as being representative of real-world conditions.

Conclusive/Inconclusive: A Conclusive Test is a valid test that has reached the desired Level of Confidence (95% is the most commonly used standard). An Inconclusive Test is a valid test that failed to reach the desired Level of Confidence for the primary KPI (95% is the most commonly used standard). Inconclusive tests, while not the marketer’s goal, are not innately bad. They offer insights into the cognitive psychology of the customer. They help marketers discover which mental levers do not have a significant impact on the decision process.

KPIs — MAIN, SECONDARY, TERTIARY

Name: KPIs are key performance indicators. They are the yardstick for measuring your test. The main KPI is what ultimately determines how well your test performed, but secondary and tertiary KPIs can be insightful as well. For example, the main KPI for a product page test might be the add-to-cart rate. That is the main action you are trying to influence with your test treatment(s). A secondary KPI might be a change in revenue. Perhaps you get fewer orders, but at a higher value per order, and thus more revenue. A tertiary KPI might be checkout rate, tracking how many people complete the action all the way through the funnel. There may be later steps in the funnel that are affecting that checkout rate beyond what you’re testing, which is why it is not the main KPI of the test but still important to understand. (Please note, every test does not necessarily have to have a main, secondary and tertiary KPI, but every test should at least have a main KPI.)

Key Discoveries: This is the main benefit of running tests — to make new discoveries about customer behavior. This Test Discovery Library gives you a central, easily accessible place to share those discoveries with the entire company. For example, you could upload this document to an internal SharePoint or intranet, or even email it around every time a test is complete.

The hypothesis will heavily inform the key discoveries section, but you may also learn something you weren’t expecting, especially from secondary KPIs.

What did the test results tell you about the perceived credibility of your product and brand? The level of brand exposure customers have previously had? Customers’ propensity to buy or become a lead? The difference in the behavior of new and returning visits to your website? The preference for different communication mechanisms (e.g., live chat vs. video chat)? Behavior on different devices (e.g., desktop vs. mobile)? These are just examples; the list could go on forever … and you likely have some that are unique to your organization.

Experience Implemented? This is pretty straightforward. Has the experience that was tested been implemented as the new landing page, home page, etc., after the test closed?

Date of implementation: If the experience has been implemented, when was it implemented? Recording this information can help you go back and make sure overall performance correlated with your expectations from the test results.

ADDING MORE TESTS TO THE TOOL

The Test Dashboard tab dynamically pulls in all information from the subsequent test worksheets, so you do not need to manually enter any data here except for the test sequence number in Column A. If you want to create a new test tab and the corresponding row in the “Test Dashboard,” follow these instructions:

    • Right click on the bottom tab titled “Template – Your Test Name Here.” Choose “Move or Copy.” From the list of sheets, choose “Template – Your Test Name Here.” Check the box “Create a Copy” and click OK. Right click on your new “Template – Your Test Name Here (2)” tab and rename as “Your Test Name Here (7).”
    • Now, you’ll need to add a new row to your “Test Dashboard” tab. Copy the last row. For example, select row 8 on the “Test Dashboard” tab, copy/paste those contents into row 9. You will need to make the following edits to reference your new tab, “Your Test Name Here (7).” This can be done in the following way:
      • Manually enter the test as “7” in cell A9.
      • The remaining cells dynamically pull the data in. However, since you copy/paste, they are still referencing the test above. To update this, highlight select row 9 again. On the Home Tab>Editing, select “Find & Select (located on the far right)>”Replace,” or use “CTRL+F”>Replace.
      • On the Replace tab of the box, enter Find What: “Your Test Name (6)” and Replace with: “Your Test Name (7).”
      • Click “Replace All”
      • All cells in the row should now reference your new tab, “Your Test Name (7)” properly.

 

Click Here to Download Your FREE Test Discovery Tool Instantly

(no form to fill out, just click to get your instant download of this Excel-based tool)

 

Special thanks to Research Manager Alissa Shaw, Data Scientist Delaney Dempsey, Associate Director of Design Lauren Leonard, Senior Director of Research Partnerships Austin McCraw, and Copy Editor Linda Johnson for helping to create the Test Discovery Library tool.

If you have any questions, you can email us at info@MECLABS.com. And here are some more resources to help with your testing …

Lead your team to breakthrough results with A Model of your Customer’s Mind: These 21 charts and tools have helped capture more than $500 million in (carefully measured) test wins

Test Planning Scenario Tool – This simple tool helps you visualize factors that affect the ROI implications of test sequencing

Customer Theory: How we learned from a previous test to drive a 40% increase in CTR

The post Get Your Free Test Discovery Tool to Help Log all the Results and Discoveries from Your Company’s Marketing Tests appeared first on MarketingExperiments.

A/B Testing: Why do different sample size calculators and testing platforms produce different estimates of statistical significance?

A/B testing is a powerful way to increase conversion (e.g., 638% more leads, 78% more conversion on a product page, etc.).

Its strength lies in its predictive ability. When you implement the alternate version suggested by the test, your conversion funnel actually performs the way the test indicated that it would.

To help determine that, you want to ensure you’re running valid tests. And before you decide to implement related changes, you want to ensure your test is conclusive and not just a result of random chance. One important element of a conclusive test is that the results show a statistically significant difference between the control and the treatment.

Many platforms will include something like a “statistical significance status” with your results to help you determine this. There are also several sample size calculators available online, and different calculators may suggest you need different sample sizes for your test.

But what do those numbers really mean? We’ll explore that topic in this MarketingExperiments article.

A word of caution for marketing and advertising creatives: This article includes several paragraphs that talk about statistics in a mathy way — and even contains a mathematical equation (in case these may pose a trigger risk for you). Even so, we’ve done our best to use them only where they serve to clarify rather than complicate.

Why does statistical significance matter?

To set the stage for talking about sample size and statistical significance, it’s worth mentioning a few words about the nature and purpose of testing (aka inferential experimentation) and the nomenclature we’ll use.

We test in order to infer some important characteristics about a whole population by observing a small subset of members from the population called a “Sample.”

MECLABS metatheory dubs a test that successfully accomplishes this purpose a “Useful” test.

The Usefulness (predictiveness) of a test is affected by two key features: “Validity” and “Conclusiveness.”

Statistical significance is one factor that helps to determine if a test is useful. A useful test is one that can be trusted to accurately reflect how the “system” will perform under real-world conditions.

Having an insufficient sample size presents a validity threat known as Sample Distortion Effect. This is a danger because if you don’t get a large enough sample size, any apparent performance differences may have been due to random variation and not true insights into your customers’ behavior. This could give you false confidence that a landing page change that you tested will improve your results if you implement it, when it actually won’t.

“Seemingly unlikely things DO sometimes happen, purely ‘by coincidence’ (aka due to random variation). Statistical methods help us to distinguish between valuable insights and worthless superstitions,” said Bob Kemper, Executive Director, Infrastructure Support Services at MECLABS Institute.

“By our very nature, humans are instinctively programmed to seek out and recognize patterns: think ‘Hmm, did you notice that the last five people who ate those purplish berries down by the river died the next day?’” he said.

A conclusive test is a valid test (There are other validity threats in addition to sample distortion effect.) that has reached a desired Level of Confidence, or LoC (95% is the most commonly used standard).

In practice, at 95% LoC, the 95% confidence interval for the difference between control and treatment rates of the key performance indicator (KPI) does not include zero.

A simple way to think of this is that a conclusive test means you are 95% confident the treatment will perform at least as well as the control on the primary KPI.  So the performance you’ll actually get, once it’s in production for all traffic, will be somewhere inside the Confidence Interval (shown in yellow above).  Determining level of confidence requires some math.

Why do different testing platforms and related tools offer such disparate estimates of required sample size? 

One of MECLABS Institute’s Research Partners who is president of an internet company recently asked our analysts about this topic. His team found a sample size calculator tool online from a reputable company and noticed how different its estimate of minimum sample size was compared to the internal tool MECLABS analysts use when working with Research Partners (MECLABS is the parent research organization of MarketingExperiments).

The simple answer is that the two tools approach the estimation problem using different assumptions and statistical models, much the way there are several competing models for predicting the path of hurricanes and tropical storms.

Living in Jacksonville, Florida, an area that is often under hurricane threats, I can tell you there’s been much debate over which among the several competing models is most accurate (and now there’s even a newer, Next Gen model). Similarly, there is debate in the optimization testing world about which statistical models are best.

The goal of this article isn’t to take sides, just to give you a closer look at why different tools produce different estimates. Not because the math is “wrong” in any of them, they simply employ different approaches.

“While the underlying philosophies supporting each differ, and they approach empirical inference in subtly different ways, both can be used profitably in marketing experimentation,” said Danitza Dragovic, Digital Optimization Specialist at MECLABS Institute.

In this case, in seeking to understand the business implications of test duration and confidence in results, it was understandably confusing for our Research Partner to see different sample size calculations based upon the tool used. It wasn’t clear that a pre-determined sample size is fundamental to testing in some calculations, while other platforms ultimately determine test results irrespective of pre-determined sample sizes, using prior probabilities assigned by the platform, and provide sample size calculators simply as a planning tool.

Let’s take a closer look at each …

Classical statistics 

The MECLABS Test Protocol employs a group of statistical methods based on the “Z-test,” arising from “classical statistics” principles that adopt a Frequentist approach, which makes predictions using only data from the current experiment.

With this method, recent traffic and performance levels are used to compute a single fixed minimum sample size before launching the test.  Status checks are made to detect any potential test setup or instrumentation problems, but LoC (level of confidence) is not computed until the test has reached the pre-established minimum sample size.

While historically the most commonly used for scientific and academic experimental research for the last century, this classical approach is now being met by theoretical and practical competition from tools that use (or incorporate) a different statistical school of thought based upon the principles of Bayesian probability theory. Though Bayesian theory is far from new (Thomas Bayes proposed its foundations more than 250 years ago), its practical application for real-time optimization research required computational speed and capacity only recently available.

Breaking Tradition: Toward optimization breakthroughs

“Among the criticisms of the traditional frequentist approach has been its counterintuitive ‘negative inference’ approach and thought process, accompanied by a correspondingly ‘backwards’ nomenclature. For instance, you don’t ‘prove your hypothesis’ (like normal people), but instead you ‘fail to reject your Null hypothesis’ — I mean, who talks (or thinks) like that?” Kemper said.

He continued, “While Bayesian probability is not without its own weird lexical contrivances (Can you say ‘posterior predictive’?), its inferential frame of reference is more consistent with the way most people naturally think, like assigning the ’probability of a hypothesis being True’ based on your past experience with such things. For a purist Frequentist, it’s impolite (indeed sacrilegious) to go into a test with a preconceived ‘favorite’ or ‘preferred answer.’ One must simply objectively conduct the test and ‘see what the data says.’ As a consequence, the statement of the findings from a typical Bayesian test — i.e., a Bayesian inference — is much more satisfying to a non-specialist in science or statistics than is an equivalent traditional/frequentist one.”

Hybrid approaches

Some platforms use a sequential likelihood ratio test that combines a Frequentist approach with a Bayesian approach. The adjective “sequential” refers to the approach’s continual recalculation of the minimum sample size for sufficiency as new data arrives, with the goal of minimizing the likelihood of a false positive arising from stopping data collection too soon.

Although an online test estimator using this method may give a rough sample size, this method was specifically designed to avoid having to rely on a predetermined sample size, or predetermined minimum effect size. Instead, the test is monitored, and the tool indicates at what point you can be confident in the results.

In many cases, this approach may result in shorter tests due to unexpectedly high effect sizes. But when tools employ proprietary methodologies, the way that minimum sample size is ultimately determined may be opaque to the marketer.

CONSIDERATIONS FOR EACH OF THESE APPROACHES

Classical “static” approaches

Classical statistical tests, such as Z-tests, are the de facto standard across a broad spectrum of industries and disciplines, including academia. They arise from the concepts of normal distribution (think bell curve) and probability theory described by mathematicians Abraham de Moivre and Carl Friedrich Gauss in the 17th to 19th centuries. (Normal distribution is also known as Gaussian distribution.)  Z-tests are commonly used in medical and social science research.

They require you to estimate the minimum detectable effect-size before launching the test and then refrain from “peeking at” Level of Confidence until the corresponding minimum sample size is reached.  For example, the MECLABS Sample Size Estimation Tool used with Research Partners requires that our analysts make pre-test estimates of:

  • The projected success rate — for example, conversion rate, clickthrough rate (CTR), etc.
  • The minimum relative difference you wish to detect — how big a difference is needed to make the test worth conducting? The greater this “effect size,” the fewer samples are needed to confidently assert that there is, in fact, an actual difference between the treatments. Of course, the smaller the design’s “minimum detectable difference,” the harder it is to achieve that threshold.
  • The statistical significance level — this is the probability of accidentally concluding there is a difference due to sampling error when really there is no difference (aka Type-I error). MECLABS recommends a five percent statistical significance which equates to a 95% desired Level of Confidence (LoC).
  • The arrival rate in terms of total arrivals per day — this would be your total estimated traffic level if you’re testing landing pages. “For example, if the element being tested is a page in your ecommerce lower funnel (shopping cart), then the ‘arrival rate’ would be the total number of visitors who click the ‘My Cart’ or ‘Buy Now’ button, entering the shopping cart section of the sales funnel and who will experience either the control or an experimental treatment of your test,” Kemper said.
  • The number of primary treatments — for example, this would be two if you’re running an A/B test with a control and one experimental treatment.

Typically, analysts draw upon a forensic data analysis conducted at the outset combined with test results measured throughout the Research Partnership to arrive at these inputs.

“Dynamic” approaches 

Dynamic, or “adaptive” sampling approaches, such as the sequential likelihood ratio test, are a more recent development and tend to incorporate methods beyond those recognized by classical statistics.

In part, these methods weren’t introduced sooner due to technical limitations. Because adaptive sampling employs frequent computational reassessment of sample size sufficiency and may even be adjusting the balance of incoming traffic among treatments, they were impractical until they could be hosted on machines with the computing capacity to keep up.

One potential benefit can be the test duration. “Under certain circumstances (for example, when actual treatment performance is very different from test-design assumptions), tests may be able to be significantly foreshortened, especially when actual treatment effects are very large,” Kemper said.

This is where prior data is so important to this approach. The model can shorten test duration specifically because it takes prior data into account. An attendant limitation is that it can be difficult to identify what prior data is used and exactly how statistical significance is calculated. This doesn’t necessarily make the math any less sound or valid, it just makes it somewhat less transparent. And the quality/applicability of the priors can be critical to the accuracy of the outcome.

As Georgi Z. Georgiev explains in Issues with Current Bayesian Approaches to A/B Testing in Conversion Rate Optimization, “An end user would be left to wonder: what prior exactly is used in the calculations? Does it concentrate probability mass around a certain point? How informative exactly is it and what weight does it have over the observed data from a particular test? How robust with regards to the data and the resulting posterior is it? Without answers to these and other questions an end user might have a hard time interpreting results.”

As with other things unique to a specific platform, it also impinges on the portability of the data, as Georgiev explains:

A practitioner who wants to do that [compare results of different tests run on different platforms] will find himself in a situation where it cannot really be done, since a test ran on one platform and ended with a given value of a statistic of interest cannot be compared to another test with the same value of a statistic of interest ran on another platform, due to the different priors involved. This makes sharing of knowledge between practitioners of such platforms significantly more difficult, if not impossible since the priors might not be known to the user.

Interpreting MECLABS (classical approach) test duration estimates 

At MECLABS, the estimated minimum required sample size for most experiments conducted with Research Partners is calculated using classical statistics. For example, the formula for computing the number of samples needed for two proportions that are evenly split (uneven splits use a different and slightly more complicated formula) is provided by:

Solving for n yields:

Variables:

  • n: the minimum number of samples required per treatment
  • z: the Z statistic value corresponding with the desired Level of Confidence
  • p: the pooled success proportion — a value between 0 – 1 — (i.e., of clicks, conversions, etc.)
  • δ: the difference of success proportions among the treatments

This formula is used for tests that have an even split among treatments.

Once “samples per treatment” (n) has been calculated, it is multiplied by the number of primary treatments being tested to estimate the minimum number of total samples required to detect the specified amount of “treatment effect” (performance lift) with at least the specified Level of Confidence, presuming the selection of test subjects is random.

The estimated test duration, typically expressed in days, is then calculated by dividing the required total sample size by the expected average traffic level, expressed as visitors per day arriving at the test.

Finding your way 

“As a marketer using experimentation to optimize your organization’s sales performance, you will find your own style and your own way to your destination,” Kemper said.

“Like travel, the path you choose depends on a variety of factors, including your skills, your priorities and your budget. Getting over the mountains, you might choose to climb, bike, drive or fly; and there are products and service providers who can assist you with each,” he advised.

Understanding sampling method and minimum required sample size will help you to choose the best path for your organization. This article is intended to provide a starting point. Take a look at the links to related articles below for further research on sample sizes in particular and testing in general.

Related Resources

17 charts and tools have helped capture more than $500 million in (carefully measured) test wins

MECLABS Institute Online Testing on-demand certification course

Marketing Optimization: How To Determine The Proper Sample Size

A/B Testing: Working With A Very Small Sample Size Is Difficult, But Not Impossible

A/B Testing: Split Tests Are Meaningless Without The Proper Sample Size

Two Factors that Affect the Validity of Your Test Estimation

Frequentist A/B test (good basic overview by Ethen Liu)

Bayesian vs Frequentist A/B Testing – What’s the Difference? (by Alex Birkett on ConversionXL)

Thinking about A/B Testing for Your Client? Read This First. (by Emīls Vēveris on Shopify)

On the scalability of statistical procedures: why the p-value bashers just don’t get it. (by Jeff Leek on SimplyStats)

Bayesian vs Frequentist Statistics (by Leonid Pekelis on Optimizely Blog)

Statistics for the Internet Age: The Story Behind Optimizely’s New Stats Engine (by Leonid Pekelis on Optimizely Blog)

Issues with Current Bayesian Approaches to A/B Testing in Conversion Rate Optimization (by Georgi Z. Georgiev on Analytics-Toolkit.com)

 

The post A/B Testing: Why do different sample size calculators and testing platforms produce different estimates of statistical significance? appeared first on MarketingExperiments.

A/B Testing Prioritization: The surprising ROI impact of test order

I want everything. And I want it now.

I’m sure you do, too.

But let me tell you about my marketing department. Resources aren’t infinite. I can’t do everything right away. I need to focus myself and my team on the right things.

Unless you found a genie in a bottle and wished for an infinite marketing budget (right after you wished for unlimited wishes, natch), I’m guessing you’re in the same boat.

When it comes to your conversion rate optimization program, it means running the most impactful tests. As Stephen Walsh said when he wrote about 19 possible A/B tests for your website on Neil Patel’s blog, “testing every random aspect of your website can often be counter-productive.”

Of course, you probably already know that. What may surprise you is this …

It’s not enough to run the right tests, you will get a higher ROI if you run them in the right order

To help you discover the optimal testing sequence for your marketing department, we’ve created the free MECLABS Institute Test Planning Scenario Tool (MECLABS is the parent research organization of MarketingExperiments).

Let’s look at a few example scenarios.

Scenario #1: Level of effort and level of impact

Tests will have different levels of effort to run. For example, it’s easier to make a simple copy change to a headline than to change a shopping cart.

This level of effort (LOE) sometimes correlates to the level of impact the test will have to your bottom line. For example, a radical redesign might be a higher LOE to launch, but it will also likely produce a higher lift than a simple, small change.

So how does the order you run a high effort, high return, and low effort, low return test sequence affect results? Again, we’re not saying choose one test over another. We’re simply talking about timing. To the test planning scenario tool …

Test 1 (Low LOE, low level of impact)

  • Business impact — 15% more revenue than the control
  • Build Time — 2 weeks

Test 2 (High LOE, high level of impact)

  • Business impact — 47% more revenue than the control
  • Build Time — 6 weeks

Let’s look at the revenue impact over a six-month period. According to the test planning tool, if the control is generating $30,000 in revenue per month, running a test where the treatment has a low LOE and a low level of impact (Test 1) first will generate $22,800 more revenue than running a test where the treatment has a high LOE and a high level of impact (Test 2) first.

Scenario #2: An even larger discrepancy in the level of impact

It can be hard to predict the exact level of business impact. So what if the business impact differential between the higher LOE test is even greater than in Scenario #1, and both treatments perform even better than they did in Scenario #1? How would test sequence affect results in that case?

Let’s run the numbers in the Test Planning Scenario Tool.

Test 1 (Low LOE, low level of impact)

  • Business impact — 25% more revenue than the control
  • Build Time — 2 weeks

Test 2 (High LOE, high level of impact)

  • Business impact — 125% more revenue than the control
  • Build Time — 6 weeks

According to the test planning tool, if the control is generating $30,000 in revenue per month, running Test 1 (low LOE, low level of impact) first will generate $45,000 more revenue than running Test 2 (high LOE, high level of impact) first.

Again, same tests (over a six-month period) just a different order. And you gain $45,000 more in revenue.

“It is particularly interesting to see the benefits of running the lower LOE and lower impact test first so that its benefits could be reaped throughout the duration of the longer development schedule on the higher LOE test. The financial impact difference — landing in the tens of thousands of dollars — may be particularly shocking to some readers,” said Rebecca Strally, Director, Optimization and Design, MECLABS Institute.

Scenario #3: Fewer development resources

In the above two examples, the tests were able to be developed simultaneously. What if the test cannot be developed simultaneously (must be developed sequentially) and can’t be developed until the previous test has been implemented? Perhaps this is because of your organization’s development methodology (Agile vs. Waterfall, etc.), or there is just simply a limit on your development resources. (They likely have many other projects than just developing your tests.)

Let’s look at that scenario, this time with three treatments.

Test 1 (Low LOE, low level of impact)

  • Business impact — 10% more revenue than the control
  • Build Time — 2 weeks

Test 2 (High LOE, high level of impact)

  • Business impact — 360% more revenue than the control
  • Build Time — 6 weeks

Test 3 (Medium LOE, medium level of impact)

  • Business impact — 70% more revenue than the control
  • Build Time — 3 weeks

In this scenario, Test 2 first, then Test 1 and finally Test 3, along with Test 2, then Test 3, then Test 1 were the highest-performing scenarios. The lowest-performing scenario was Test 3, Test 1, Test 2. The difference was $894,000 more revenue from using one of the highest-performing test sequences versus the lowest-performing test sequence.

“If development for tests could not take place simultaneously, there would be a bigger discrepancy in overall revenue from different test sequences,” Strally said.

“Running a higher LOE test first suddenly has a much larger financial payoff. This is notable because once the largest impact has been achieved, it doesn’t matter in what order the smaller LOE and impact tests are run, the final dollar amounts are the same. Development limitations (although I’ve rarely seen them this extreme in the real world) created a situation where whichever test went first had a much longer opportunity to impact the final financial numbers. The added front time certainly helped to push running the highest LOE and impact test first to the front of the financial pack,” she added.

The Next Scenario Is Up To You: Now forecast your own most profitable test sequences

You likely don’t have the exact perfect information we provided in the scenarios. We’ve provided model scenarios above, but the real world can be trickier. After all, as Nobel Prize-winning physicist Niels Bohr said, “Prediction is very difficult, especially if it’s about the future.”

“We rarely have this level of information about the possible financial impact of a test prior to development and launch when working to optimize conversion for MECLABS Research Partners. At best, the team often only has a general guess as to the level of impact expected, and it’s rarely translated into a dollar amount,” Strally said.

That’s why we’re providing the Test Planning Scenario Tool as a free, instant download. It’s easy to run a few different scenarios in the tool based on different levels of projected results and see how the test order can affect overall revenue. You can then use the visual charts and numbers created by the tool to make the case to your team, clients and business leaders about what order you should run your company’s tests.

Don’t put your tests on autopilot

Of course, things don’t always go according to plan. This tool is just a start. To have a successful conversion optimization practice, you have to actively monitor your tests and advocate for the results because there are a number of additional items that could impact an optimal testing sequence.

“There’s also the reality of testing which is not represented in these very clean charts. For example, things like validity threats popping up midtest and causing a longer run time, treatments not being possible to implement, and Research Partners requesting changes to winning treatments after the results are in, all take place regularly and would greatly shift the timing and financial implications of any testing sequence,” Strally said.

“In reality though, the number one risk to a preplanned DOE (design of experiments) in my experience is an unplanned result. I don’t mean the control winning when we thought the treatment would outperform. I mean a test coming back a winner in the main KPI (key performance indicator) with an unexpected customer insight result, or an insignificant result coming back with odd customer behavior data. This type of result often creates a longer analysis period and the need to go back to the drawing board to develop a test that will answer a question we didn’t even know we needed to ask. We are often highly invested in getting these answers because of their long-term positive impact potential and will pause all other work — lowering financial impact — to get these questions answered to our satisfaction,” she said.

Related Resources

MECLABS Institute Online Testing on-demand certification course

Offline and Online Optimization: Cabela’s shares tactics from 51 years of offline testing, 7 years of digital testing

Landing Page Testing: Designing And Prioritizing Experiments

Email Optimization: How To Prioritize Your A/B Testing

The post A/B Testing Prioritization: The surprising ROI impact of test order appeared first on MarketingExperiments.

In Conversion Optimization, The Loser Takes It All

Most of us at some point in our lives have experienced that creeping, irrational fear of failure, of being an imposter in our chosen profession or deemed “a Loser” for not getting something right the first time. marketers who work in A/B testing and conversion optimization.

We are constantly tasked with creating new, better experiences for our company or client and in turn the customers they serve. Yet unlike many business ventures or fire-and-forget ad agency work, we then willingly set out to definitively prove that our new version is better than the old, thus throwing ourselves upon the dual fates of customer decision making and statistical significance.

And that’s when the sense of failure begins to creep in, when you have to present a losing test to well-meaning clients or peers who were so convinced that this was a winner, a surefire hit. The initial illusion they had — that you knew all the right answers — so clinically shattered by that negative percentage sign in front of your results.

Yet of course herein lays the mistake of both the client and peer who understandably need quick, short-term results or the bravado of the marketer who thinks they can always get it right the first time.

A/B testing and conversion optimization, like the scientific method these disciplines apply to marketing, is merely a process to get you to the right answer, and to view it as the answer itself is to mistake the map for the territory.

I was reminded of this the other day when listening to one of my favorite science podcasts, “The Skeptics Guide to the Universe,” hosted by Dr. Steven Novella, which ends each week with a relevant quote. That week they quoted Brazilian-born, English, Nobel Prize-winning zoologist Sir Peter B. Medawar (1915 -1987) from his 1979 book “Advice to a Young Scientist.” In it he stated, “All experimentation is criticism. If an experiment does not hold out the possibility of causing one to revise one’s views, it is hard to see why it should be done at all.”This quote for me captures a lot of the truisms I’ve learnt in my experience as a conversion optimization marketer, as well as addresses a lot of the confusion in many MECLABS Institute Research Partners and colleagues who are less familiar with the nature and process of conversion optimization.

Here are four points to keep in mind if you choose to take a scientific approach to your marketing:

1. If you truly knew what the best customer experience was, then you wouldn’t test

I have previously been asked after presenting a thoroughly researched outline of planned testing, that although the methodic process to learning we had just outlined was greatly appreciated, did we not know a shortcut we could take to get to a big success.

Now, this is a fully understandable sentiment, especially in the business world where time is money and everyone needs to meet their targets yesterday. That said, the question does fundamentally miss the value of conversion optimizing testing, if not the value of the scientific method itself. Remember, this method of inquiry has allowed us — through experimentation and the repeated failure of educated, but ultimately false hypotheses — to finally develop the correct hypothesis and understanding of the available facts. As a result, we are able to cure disease, put humans on the moon and develop better-converting landing pages.

In the same vein, as marketers we can do in-depth data and customer research to get us closer to identifying the correct conversion problems in a marketing funnel and to work out strong hypotheses about what the best solutions are, but ultimately we can’t know the true answer until we test it.

A genuine scientific experiment should be trying to prove itself wrong as much as it is proving itself right. It is only through testing out our false hypothesis that we as marketers can confirm the true hypothesis that represents the correct interpretation of the available data and understanding of our customers that will allow us to get the big success we seek for our clients and customers.

2. If you know the answer, just implement it

This particularly applies to broken elements in your marketing or conversion funnel.

An example of this from my own recent experience with a client was when we noticed in our initial forensic conversion analysis of their site that the design of their cart made it almost impossible to convert on a small mobile or desktop screen if you had more than two products in your cart.

Looking at the data and the results from our own user testing, we could see that this was clearly broken and not just an underperformance. So we just recommended that they fix it, which they did.

We were then able to move on and optimize the now-functioning cart and lower funnel through testing, rather than wasting everyone’s time with a test that was a foregone conclusion.

3. If you see no compelling reason why a potential test would change customer behavior, then don’t do it

When creating the hypothesis (the supposition that can be supported or refuted via the outcome of your test), make sure it is a hypothesis based upon an interpretation of available evidence and a theory about your customer.

Running the test should teach you something about both your interpretation of the data and the empathetic understanding you think you have of your customer.

If running the test will do neither, then it is unlikely to be impactful and probably not worth running.

4. Make sure that the changes you make are big enough and loud enough to impact customer behavior

You might have data to support the changes in your treatment and a well-thought-out customer theory, but if the changes you make are implemented in a way that customers won’t notice them, then you are unlikely to elicit the change you expect to see and have no possibility of learning something.

Failure is a feature, not a bug

So next time you are feeling like a loser, when you are trying to explain why your conversion optimization test lost:

  • Remind your audience that educated failure is an intentional part of the process:
  • Focus on what you learnt about your customer and how you have improved upon your initial understanding of the data.
  • Explain how you helped the client avoid implementing the initial “winning idea” that, it turns out, wasn’t such a winner — and all the money this saved them.

Remember, like all scientific testing, conversion optimization might be slow, methodical and paved with losing tests, but it is ultimately the only guaranteed way to build repeatable, iterative, transferable success across a business.

Related Resources:

Optimizing Headlines & Subject Lines

Consumer Reports Value Proposition Test: What You Can Learn From A 29% Drop In Clickthrough

MarketingExperiments Research Journal (Q1 2011) — See “Landing Page Optimization: Identifying friction to increase conversion and win a Nobel Prize” starting on page 106

The post In Conversion Optimization, The Loser Takes It All appeared first on MarketingExperiments.