Before you engage in CRO ideas that drive sales opportunities deeper into the funnel, many travel brands need to get customers from an email to their site. From our research into building a model of the customer’s mind, we’ve discovered a seemingly simple yet often overlooked email marketing tactic. (I’ve used three experiments from the travel industry as an example, but this discovery is broadly applicable to most industries).
Move your main call-to-action to the top section of your email, and make sure it’s clear and prominent
Remember, the goal isn’t to get the vacation package booking within your email — that’s the website’s job. Your goal is to get the click. As marketers, we often want to throw the kitchen sink at our prospects to spin the odds in our favor. I mean, they are bound to click on something at that point … right?
Wrong! Make sure your objective in your email is focused, and construct your calls-to-action accordingly. Too many options can overwhelm or confuse your prospects when they don’t know what to do. When your prospect doesn’t know what to do, they tend to abandon the email altogether.
340% increase in CTR by adding prominent CTA
Take an experiment we helped a river cruise company run. Its email had many “clickable” areas driving to the landing page, but it lacked a clear call-to-action (CTA) to invite the prospective traveler to take action. Without a prominently emphasized CTA, it is difficult for a reader to quickly identify a primary objective of this email communication.
By adding a yellow “See Offer Details” button near the top of the email that didn’t otherwise exist previously, we saw a 340% increase in clickthrough rate (CTR).
17% increase in CTR by moving CTA button to the top
In a similar test, a vacation brand had an email that actually did have a clear call to action, but it was hidden under a paragraph of copy mid-way through the email. By moving the button to the top and pulling out the most important value claims for the vacation, this version resulted in a 17% increase in clickthrough rate.
43% increase in CTR from reducing number of CTAs
Finally, here’s another A/B test example that we ran with another vacation provider where we saw a 43% lift in clickthrough rate. We reduced the number of CTAs and focused it on the key actions we wanted the prospect to take. We also made them much more visible to a prospect who is skimming dozens of emails. We continued to refine this tactic in two additional follow-up tests to fully optimize the email, which continued to compound the lift in clickthrough rate.
Bottom Line: Make the CTA a no-brainer
In email, we must recognize that our prospects skim dozens of emails in their inbox for flight deals, cruises, hotels and vacation packages. If you’re skilled enough to get the “open” from a compelling subject line, make the next micro-yes a no-brainer for your prospect.
There are many complex things you can do with email marketing from a technological and personalization perspective. But before you dive into those, the lowest-hanging fruit may be to simply test the clarity of the email “ask.”
Email Messaging on-demand certification course (from MECLABS Institute, MarketingExperiments’ parent research organization) — Take this course to capture more subscribers, craft effective email copy and convert email clicks to sales
Expressing value is the core of marketing. But you shouldn’t express the same value messages at every point of the customer journey or to every customer.
Different people need to understand different elements of your product or service’s value at different points.
Complex, I know. But let’s try to have a little fun with it.
To help you create a primary testing focus that will allow you to better understand the most effective value sequencing for your website, we created an entertaining visual tool — the “What kind of value prop should you add to your webpage?” decider graphic. (Why should “Which Kardashian are you?” or “What is most likely hiding under your date’s couch cushions?” have all the fun?)
Our goal is to provide a visual flowchart to help you flex those customer curiosity muscles and understand the theory of value sequencing during the customer journey. And hopefully, we’ll inspire some impactful test ideas along the way.
And feel free to whip out your own Expo markers and draw a possible value sequencing decider graphic for your unique customers on your office whiteboard (if you do, we’d love to see it. Just mention @MktgExperiments on Twitter when you share it).
START BY PICKING YOUR WEBPAGE …
Does the purchase decision occur on the webpage you are working on? Marketers can easily fall into the trap of sell, sell — that is, selling a product the same way at every single customer touchpoint. But that’s not always what the customer needs or wants. Every website — whether B2B or B2C, whether the product is a set of steak knives or a white paper download — generally has pages created specifically to sell the product.
Every other page happens either before (e.g., homepage or category page) or after (e.g., payment page or cart page) that purchase decision. When you think of value sequencing, you should not express the same value across each of these pages.
That would be like running into a friend you haven’t seen in a long time, and saying, “Hey Todd, how have you been?”
After Todd answers, you then respond, “Hey Todd, how have you been?” He then gives you a quizzical look to which you respond, “Hey Todd, how have you been?”
That would be an annoying conversation. Your customers feel the same way and will back away from your website as quickly as Todd does from your conversation if you don’t sequence the value correctly across these types of pages.
If the page is before the purchase decision
Just like a journey in the physical world, a customer journey has momentum. That motion isn’t physical though, it involves mental actions. In other words, cognitive momentum.
Think about when you go to a nice restaurant. The maître d’ doesn’t hold out his hands and yell, “17 BUCKS FOR THE SOUP!” He welcomes you in, asks about your preferences, complements your wardrobe, directs you to a table, gives you menus, tells you about your waiter, etc.
In other words, he directs your cognitive momentum. He’s not trying to sell you yet, but he likely is smoothly prepping the sale by articulating the top value claims of the overall restaurant. “Our chef just received another Michelin star, and we are so very proud.”
This is the same thing your pages should do before a purchase decision — you want to direct the cognitive momentum to the place that will best serve the customer.
For a homepage — the front door of a website — in addition to that clearly expressed primary value proposition, you want to create a process-level value proposition to get the customer to the most effective destination.
On the category page, this gets even more granular. You should help customers get to the right page (and often identify the right product) for themselves. So emphasize unique product-level value props to help customers identify the best product fit. When there are different product options, you might want to use a configurator to help clarify the value. When the different product offerings are different levels of the same essential product, make sure there are clear value gulfs between the offers.
If the purchase decision is on the page
When the purchase decision is on that specific webpage or landing page, you want to boost the cognitive momentum. Whatever brought them to this page was strong enough to get them to take the action to visit the page (clicking through an email, clicking from the homepage, etc.).
Now you must boost customers’ cognitive momentum to get them to the point of actually acting and making the purchase decision.
In general, this is where you want to express the full value proposition with images, copy, etc. (please note: there are exceptions based on your target audience, which we’ll cover next). This is your central, core conversion point. You want customers to experience the clearest expression of product value to overcome the cost they will have to pay to get it.
For a product display page, focus tightly around the product-level value prop to encourage conversion. The only expressions of the overall company value proposition should be in support of building credibility for the product-level value prop. One mistake some companies make is bragging too much about the overall company, and not focusing tightly on what matters most to the customer about the specific product.
Another key page where a decision occurs is a landing page. In this case, we use the definition of “purchase a product” a little more broadly. For example, the product might be a content download and the purchase price may be filling out an information form, not necessarily paying with money.
Regardless, a key conversion is happening, and it is essential to boost cognitive momentum. For a landing page, stay focused on communicating value in line with the page’s objective and reduce distractions from that key value. That doesn’t necessarily mean the page has to be short. Long landing pages can convert better than shorter ones when the value is focused around the key conversion objective.
If the page is after the purchase decision
Depending on your funnel, getting a “yes” on the key purchase decision may not be enough. Don’t overlook the pages that come after that decision.
This doesn’t mean you need to continue selling the customer like you did on the purchase decision page. But you don’t want to assume the decision is final either.
After the key purchase decision, you need to support cognitive momentum until a final conversion is made (and, really, even after that final decision). To do so, just remind them of value while reducing friction on the payment page. Or reinforce the product-level value proposition in the shopping cart.
But whatever you do, don’t take the sale for granted.
… THEN DEFINE YOUR TARGET AUDIENCE
All of the above is generally applicable to a general customer. To make your value sequencing even more effective, consider the specific type of customer you’re talking to by building a segmentation strategy that leverages technology — like marketing automation, a customer data platform, a data management platform, and/or a customer relationship management platform. This will help personalize that value communication.
When to focus on getting out of their way
Previous customers understand the value of your company and its products on a level that prospects don’t because they have already experienced it firsthand. How you communicate to them is determined by the level they have experienced it.
If the customer has already purchased that product from that page before, testing should focus primarily on the most effective approach to just get the heck out of their way. Forget value communication, they already understand the value. Simply reduce friction and anxiety.
Amazon is a master at this technique. From the “Buy it Again” button and “Subscribe & Save” to off-page optimization tactics like the Dash button and voice shopping with Alexa, Amazon has mastered the tactic of getting out of repeat customers’ way and making it easy for them to make another purchase.
You should also test getting out of their way if you have reason to believe that customers understand the exclusive value of your offering already.
These could be prospects that are familiar with your brand, and you have reason to believe they’ve heavily researched your product beforehand and understand its value — repeat visitors who are customers of your content and have just been waiting for the right time to buy, or customers that have bought other products from your brand and understand the value of this product type. Whatever the exact path, if you have reason to believe customers already understand the value of your offering and company, value communication can sometimes decrease conversion. Test the idea of getting out of their way and see what you discover about your customer. It will either affirm your belief that they already understand the value, or help you build a more robust customer theory around where they are in the buying process and what value they need to understand.
When to focus on paving the way
If the customer has bought the product before but through another channel, say, a brick-and-mortar store, for example, you simply want to pave the way for them to purchase on this page. Focus on communicating the process-level value prop and reducing friction so they understand why it is worth making the purchase through your website while making it easy for them.
Another great example of when you want to take the paving-the-way approach is transitioning a customer from purchasing your product(s) through a third-party marketplace to purchasing on your own e-commerce store (Pro tip: Understand the marketplace’s terms and conditions and don’t violate them).
When to focus on making them informed
Whether customers have bought from your brand before or not, you need to determine how familiar they are with the product type. Let’s take an example that isn’t a typical ecommerce purchase, to exaggerate the value communication that is necessary. If the landing page is for a Nissan LEAF, you need to inform them of the value of an electric vehicle. You should sell the category while selling the car — communicate the category’s (in this case, electric vehicles) value prop, in addition to the product-level value proposition and the process-level value proposition.
If the product type isn’t different enough, don’t assume they understand the value. They need to understand that category value before (or at least while) they are assessing the value of your specific product.
This use case is especially prevalent when you have an innovative product. Whether it’s cloud computing or flying cars (“Back to the Future Part II” foretold AI-driven voice assistants, so you know flying cars are just around the corner), most people won’t buy the product until they buy into the category. Communicating category value correctly will also increase the likelihood of satisfied customers (because they know what they’re getting into along with the limitations of the new technology) and positive word-of-mouth that speeds widespread adoption. If the true value of the new innovation isn’t clearly communicated (or there to begin with), the word-of-mouth could kill a nascent technology before it has the chance to blossom.
When to focus on making it clear
If customers are familiar with your brand and understand the value the company provides, but don’t understand the exclusive value of your offering, take an approach that will help make it clear. Communicate the product- and process-level value prop. You want to especially leverage exclusivity in that product-level value prop, so customers can easily understand the key differentiator between what you offer and what the competitor offers.
For example, if I understand the value of two different electronics companies, those will both be in my decision set when I am shopping for headphones. But as I get deeper into my customer journey, I discover only one of the products has a lifetime warranty. That is the one I will likely choose. By understanding the value both companies bring, I trust them both, and they have both made it into my decision set. But the exclusive value for the specific product set it apart and ultimately led me to choose it over the competition.
When to focus on relieving anxiety
If customers don’t understand the value of your company, you usually want to at least communicate some level of your brand’s primary value proposition along with a product- and process-level value prop. But space and time are limited, so what aspects of the primary value prop should you focus on? The risk level of the purchase should inform that focus. If a product is high risk — for example, it is expensive or customers are worried it’s a safe product — relieving anxiety is a key element to focus on.
Emphasize the credibility aspect of the company’s value proposition while communicating the product- and process-level value prop. For example, if it’s a life insurance company you likely not only want to communicate product details, but also message how long the company has been in business and how financially solid it is. Most customers will not buy even the most appealing life insurance product (or vitamins or a bet-the-business-on-it software platform) from a company they don’t trust.
When to focus on looking more appealing
If prospective customers don’t understand the company’s value but the purchase is low risk, you could focus on testing ways to make the company look more appealing. The element of the company’s value proposition you should most emphasize is appeal, in addition to communicating the product- and process-level value proposition. When risk is low, you can tie in with the positive elements of the company’s value prop without having to work to overcome concerns as much.
At this one ad agency I knew of, their main copywriter was one of the few people who didn’t work in the building. Because he wasn’t really part of the team. Account executives would just send over some information. And he would send back the written copy.
Now, copywriters can certainly be successful while working remotely. Many do.
But nailing that conversion copy takes much more than processing a marketing brief and turning it into copy. It requires digging into the subject matter and finding that gold nugget that helps the customer truly perceive the product’s value. You need intimacy with the product and its customers.
That can also require digging for information beyond the marketing brief. And one way to get that info is to get out of your writing cave and interview a few key people. Interviews won’t give you the rigorous behavioral data that you can get from an A/B split test. But they can help you increase the intimacy with the product and its potential customers to improve your marketing copy.
Here are some ideas for questions you can ask key people. All of the questions are directional and should merely guide your conversation. Listen for key bits of information and ask follow-up questions to dive deeper anytime someone makes a vague statement like “Product X is better than Product Y” (How is it better?) or “The ads for Product X were misleading” (What about them was misleading?). And if you’re extroverted, please keep in mind your interview subjects should do way more talking than you.
And if you’re an inexperienced interviewer, don’t overlook the most powerful thing you can say … nothing. Most people naturally want to fill an awkward pause, and often provide very helpful information to fill the dead air as they think deeper. So leave plenty of room for pauses between questions as you take notes on what they say.
Company (and Key Partner) Employees
There is a wealth of wisdom within the company itself and from your key vendors and other partners. From an operations manager to quality control folks, from researchers and product developers to the very marketing manager who originally gave you the brief, company employees understand the value of the product or service being offered at a very deep level. And this can hold true for key partners as well — like outsource manufacturing or product support.
Much of the value these employees are aware of may never make it into a marketing brief. By interviewing them directly, you can minimize the disconnect between the perceived value communicated by the company’s marketing messaging and the actual value being delivered by the company.
I’m often surprised by just how big that disconnect can be. When I meet an employee of a company and ask them questions about what their company does (whether in a professional setting or at a dinner party), and then see that company’s advertising and marketing — many times I’ve thought, “Wow, they’re just not getting the value across at all.”
A great example is this article from Target Marketing magazine, where Denny Hatch shows how The New York Times obituary of Amar G. Bose was better at communicating the value of the product than the copy message in Bose ads.
The best place to dig for this value is in a value proposition workshop. But you also want to get some of this information from people who can’t attend the workshop.
Some questions to ask internally:
Why did we launch this product?
What work/research went into creating this product?
What customer need does the product serve?
How is the product better than competitors?
Why should anyone believe us? What proof points do we have to make that case?
How is it the same? Worse?
How would competitors answer that same question — how is the product better than, the same as, or worse than other options in the marketplace?
What type of customers does the product serve well?
What type of customers does the product not serve well?
What is the most interesting fact about this product that most people don’t know?
What question did I not ask you but should have?
Anything else to add? (don’t overlook the importance of this question, sometimes it produces your best answers)
Key information to listen for: People tend to write and speak differently.
When a marketer is crafting a written brief to send to a creative team, even if they’re not trying to, they tend to write more hype-filled copy. By just talking to them about the product, you will tend to get more straightforward answers. You want conversations based on facts, not hype. Challenge any statement an employee makes that sounds like hype and ask them to back it up with a data point so you can write credible copy.
One benefit of interviewing customers is that you forge a relationship with someone you can write directly to.
Powerful conversion copy isn’t written to a generic audience. It’s written with a specific person in mind, with their needs and wants, hopes and fears staring you right in the face. With their speaking cadence and reading level driving your word choices. A budget hotel is “clean” while a luxury hotel is “exquisite.” The same electric car model has “ludicrous acceleration” to a sports car shopper but “zero emissions” to an eco-conscious shopper.
Keep in mind, the feedback will be a bit skewed since you’re only talking to people who bought, not those who didn’t buy. Also, people don’t always realize why they act the way they do. As bestselling author Chip Heath has said, “people often decide based on emotion and backfill with logic.” With that understanding in mind, interviewing customers can be one input that helps you better understand and communicate with them.
Some questions to ask customers:
Do you have a set of challenges around [whatever the product does]? Please describe those challenges.
Why did you start shopping for this product?
What research did you do when shopping? What were your most trusted sources of information?
If you used a search engine, what words did you type in?
Where else did you shop?
What was better about those other products? Worse?
What is better about our product? Worse?
How do you use the product?
Now that you own the product (or have used the service), how did it compare to your expectations? What is better than what you expected? Worse?
Would you recommend this product? If so, to who specifically? What would you tell them about the product?
What question did I not ask you but should have?
Anything else to add?
Key information to listen for: It’s not just what customers say, it’s how they say it. You can hear some interesting sentiment from customers. But listen to their exact wording as well. Marketers and brands can sometimes use internally focused language or industry-heavy jargon when talking about their products. What words do your customers use? Hearing straight from the source can help you speak their language when writing copy for them. These subtle nuances in your copy can subconsciously let customers know that your product is for people like them.
It isn’t always possible to interview customers directly. And even when you can, customers may not truly understand their own motivations. So it helps to get another view of customers by monitoring the human interface between your company and your customers. Some examples of customer-facing employees are customer service, sales reps or store associates.
These interviews can also help you get information about customers who didn’t buy. A big caveat here: The reason customers tell a company employee why they didn’t buy isn’t always the same as the reason they actually didn’t buy. For example, they may not want to admit to the sales rep that he just rubbed them the wrong way. Or if they’re getting too much pushback about a particular concern from the rep, they just might not want to give any other reasons to get further hounded. So when you ask customer-facing employees about customers that didn’t buy, just take the info with a grant of salt.
Some questions to ask customer-facing employees:
What is different between our least satisfied and most satisfied customers?
What praise do you most frequently hear?
What complaints do you most frequently hear?
Have you come across any misunderstood value the customers or potential customers didn’t know the product had? (e.g., a feature they overlooked, a company policy that wasn’t clear, but they really valued, etc.)
What specific words do customers use to explain our product?
How do you describe our products to other experts in our industry?
How do you describe our products to people who aren’t in our industry? (friends, family, etc.)
What do you think of our marketing campaigns? What are they doing well? Poorly?
What do customers or potential customers give you the most pushback about?
What question did I not ask you but should have?
Anything else to add?
Key information to listen for: Customer-facing employees often have a “rubber meets the road” view on your company’s marketing campaigns and overall business vision and plan. They can help you identify the gaps in customer communication.
In-person team training — Identifying and crafting effective messaging, building landing pages that convert, and other topics to set your team up for success
I recently discovered that the tires on my Nissan LEAF were recalled, not the specific tires on my car, just that tire model. And it turns out, only the tires manufactured between February 5th and February 18th in a specific plant of that model were included in the recall.
That is impressive. In general, tires are manufactured with such repeatable high quality, that defects can be pinpointed to just a 13-day span among years and years of tire production.
Marketing is not nearly as consistent.
One way to improve the consistency of your marketing is with a repeatable methodology. And if you’re a repeat reader of MarketingExperiments, I’m sure you’re familiar with the MECLABS Institute Conversion Sequence heuristic which can bring structure, clarity and a repeatable framework to any marketing conversion goal you have (MECLABS is the parent research organization of MarketingExperiments).
This is more than just a tool you can use on landing pages. In fact, people around MECLABS have discussed using it to get their children to eat healthy. I’ve used it on college recruiting trips to help students understand elements to consider when choosing a first job.
Since its introduction more than a decade ago, we’ve written and talked about the heuristic a lot on MarketingExperiments. But others have as well. So let’s take a look at some advice from around the web suggesting ways to increase conversion — whatever your conversion goal may be — with this repeatable methodology:
“Having a quality value proposition is vital for a website. Researchers at MarketingExperiments concluded that value proposition is key to your conversion rate. Using its ‘conversion heuristic,’ they found that value proposition was second in importance, just behind a consumer’s motivation when visiting your website.”
“While we keep advising marketers to test with their specific audiences, there are actually a few best practices you should take into consideration. In fact, the folks at MECLABS came up with a formula to create top-performing landing pages.”
“One of the most popular conversion optimization heuristics is an equation. MarketingExperiments calls it a sequence. You could call it a shortcut. It summarizes the main factors in the conversion process.”
“The conversion heuristic developed by MECLABS Institute is interesting. By definition, a heuristic is a problem-solving approach which concedes that an optimal, logical, and certainly exact solution isn’t possible. Heuristics use guesstimates; measurements are often rule of thumb.”
“Each landing page should be targeted to a specific segment of your customer base, meaning there’s no exact science to a perfect landing page optimization. But our statistician friends over at MECLABS have come pretty close. They’ve developed a formula for creating an optimized landing page for any marketing campaign.”
“If you’re not a mathematician, don’t freak out, as this is not a problem you solve in the traditional sense. It’s a heuristic problem, meaning it’s a more concrete way to look at an abstract concept, such as the way we make decisions.”
“As with all marketing functions, landing page optimization is a constant work in progress. We don’t learn until we test and test again and sometimes it’s useful to have a mathematical formula assisting an otherwise creative process.”
“To think through the fundamentals of what makes a successful landing page I think this formula developed by Flint McGlaughlin and team at Marketing Experiments is great. We use it in the latest update to our Guide to Landing Page Optimisation to set the scene. We really like the way it simplifies the whole interplay between what the landing page needs to achieve for the business and what the visitor is seeking.”
“Today I want to look at motivation from a different angle. I want you to choose a landing page that is top priority for you to optimize. For example, your most profitable product with the highest abandonment rate. I want to get you thinking about which customer motivations are most likely to match your business, your products, your typical customer and your landing page presentation.”
And of course, we’ve written about the Conversion Sequence heuristic as well …
Mobile adds a challenging layer of complexity when optimizing your online marketing funnel. Testing a winning desktop experience can often produce the opposite effect on mobile performance, leaving us to diagnose where and why our good idea went wrong.
We know that users interact differently on mobile and that there are slightly differing variables at play. Yet with the rapid evolution of the space, the exact nature of those differences are still being discovered. Below you’ll find some of the mobile discoveries we’ve made in the last year of testing. And to help you put these discoveries into action, we’ve created the free MECLABS Mobile Micro Course.
Discovery #1. Design for your customer
On the extreme end, many webpages and offers try to accomplish too much. The eye path is cluttered with competing ads and calls-to-action, a saturation of colors and images, and too many boxes and options — all of which distract the user. Another extreme is when marketers fail to address critical questions in the mind of the consumer. Webpage space is peppered with hero images, irrelevant art, artistic code and design and generic qualitative claims.
For desktop, and especially mobile since concision is key (as discussed later in this article), a webpage’s objective(s) is the barometer you use to measure the relevance of your design and page elements. The goal of your page should be narrow enough to avoid confusing and overwhelming the customer. Yet your design and copy should contain enough relevant value to keep users engaged and progressing toward a macro-conversion.
Design and copy can showcase our style, but ultimately, it’s all for the customer. A webpage should be designed to perform, not to impress. Usability and clarity trump clever design every time. This is not to say that page design is not important, but that each element of the design should support and contribute to the core message — and that core message itself is the sum of your perceived value proposition.
If a significant portion of your traffic is mobile, then it is important to treat mobile as a separate experience since users almost always behave to some degree differently than on desktop. If your traffic is predominantly mobile, then don’t treat mobile as the secondary experience; treat it as the primary. Usability is even more important for keeping consumers engaged in mobile since mobile is a compressed customer experience.
“If mobile is your primary audience, always design mobile first. Your mobile audience should not get a lesser experience if they are your primary [or significant] audience,” said Jonathan Yates, Digital Marketing Lead at MECLABS Institute.
Discovery #2. Long-form is not your enemy, irrelevance is
Take a look at this example from a test conducted for a well-known university seeking to increase enrollment:
In our analysis of the control page, we determined that while customers are motivated and interested, the messaging and sequence of information fails to provide a clear, controlled thought sequence that matches the key questions in the mind of the consumer.
To optimize the thought sequence on this page, not only was the information resequenced to better match the user’s thought sequence but against many mobile best practices, information was added to create a longer-form page.
While marketers often shy away from information-heavy mobile design, it isn’t the amount of information that influences user engagement, but rather the perceived relevance of that information. It is far more important to understand the necessary sequence of questions and conclusions in the mind of the user than to adhere to rigid best-practice-rules.
There is an inverse relationship between the complexity of a product or service and the amount of information required to make a purchase a decision. When communicating a more simple, transactional product, less information might be sufficient. But failing to provide enough information can be equally detrimental to conversion as overwhelming the customer.
In this case, the longer-form version with revised messaging produced a 32% increase in conversion rate.
“Don’t be afraid to have a longer page. If the content is valuable and relevant, prospects will read it. Just remember to make it clear …” Yates said. (He led the marketing experiment above.)
Discovery #3. [BUT] Concision is critical
While the amount of information should be informed by the questions and micro-decisions on the part of the consumer, it is the marketer’s responsibility to make that information as easy as possible to consume.
In the test above, more information was added but also translated from desktop-style long-form paragraphs to simple, easy bullet points with clear headlines and visuals to guide the user. Put simply, the customer should inform what and how much information is needed, but the marketer should design the information in the clearest, simplest and easiest format for the customer.
Mobile is a compressed customer experience. On the desktop, a user has greater autonomy over how they choose to experience a page since the eye path can vary due to screen-size and design. However, in mobile, the customer experience is necessarily sequential and linear — meaning that the customer experiences one page element at a time instead of multiple competing elements. This is why it is critical to ensure that the information presented is relevant, easy-to-digest, and aligned with the customer’s sequence of questions and conclusions.
Countless marketing mistakes are made because of assumptions based on insufficient data or rationale, and every marketer should be skeptical toward the litany of “best practices” and noisy ideas in the marketplace. Mobile shopping and browsing continue to grow and become ever more important, and the only way to truly understand it is through testing.
Understanding always trumps ideas over time. There are common variables and important differences between your desktop and mobile user behavior that vary depending on your product, model, industry and audience. So, to be truly effective in mobile, we must treat mobile as different and seek to understand how, to what degree and why.
Stay tuned for more research and insights as we continue to learn about what works in mobile marketing and why.
Mind reading is not part of a marketer’s job description, but mind mapping should be.
Why? Because building a mental roadmap of the thinking process during the buyer’s journey is crucial to achieving maximum conversion, and this can only be accomplished through a rigorous testing program. There are no shortcuts.
The articulation of this roadmap must be simple and precise. We call this coherence. Flint McGlaughlin explains more:
If you haven’t heard us say it before, we will say it again: A good idea in a brainstorming session is insufficient. You must generate more than one hypothesis from an idea and then determine which one/s should be tested. This intersection of science and art helps you achieve coherence in your understanding of customer behavior. Watch now:
Lastly, Flint gives a tip on avoiding incoherence in your testing efforts: Why using “and” in your hypothesis can weaken its effectiveness.
If you’d like to get better business results and learn more about MECLABS methodology that has helped capture more than $500 million in test wins, visit our Research Services page.
But they have also led to several page templates we use regularly to achieve wins inside of companies in every industry and of every business model.
Landing Page Template
One page that is generally consistent in almost any business is the main offer page for a product or service. This “landing page,” as it is often called, accounts for the main idea behind the offer.
After thousands of a/b tests, we’ve meticulously put together a template that draws from a meta-analysis of these tests. We examined the patterns common among most landing page tests we’ve run that have achieved a business result. What we found was a series of common denominators that we then integrated into our offer page template.
We’ve also added explanations into the template on how to conform it to meet your own marketing needs. For example:
Headline and Sub-Headline
An online interaction with a prospect is like a conversation with a potential love interest. You hopefully do not begin the conversation with, “I am available for dates. Here is my number. Call me.” That is too vague and offers no reason to actually call — also it is rude.
You need a pick-up line that clearly communicates your value proposition and a sub-headline that further delineates that value proposition and how this page helps the prospect obtain that value.
If your page is going to include an image, which is worth a test, the image must be instantly recognizable and reinforce the value proposition raised in the headline and sub-headline.
It must not tax the prospect’s mental faculties trying to make sense of the image. You do not want to slow down their cognitive momentum.
Primary Information Column
This is where your main body copy goes, including some easy-to-scan bullet points. You want the reader to be able to skim through this copy and pick out the main details they need to come to an informed purchase decision.
It is one column because multiple columns of vital information disperse attention and confuse. One column creates simplicity and velocity toward the call-to-action.
One Emphasized Call-to-Action
The call-to-action should emphasize in the actual wording the value proposition of the offer. In other words, “Get Instant Access Now” is preferable to “Click Here.”
There should not be multiple equally weighted calls-to-action because this forces the prospect to weigh the options, which decreases momentum and often stalls purchase intent.
The primary information column will not suffice for some prospects. They will want more information before coming to a decision.
For these prospects, you can include additional information below the call-to-action. Supporting content can also include elements like testimonials, trust logos and additional copy and images.
Also, if you would like to get the other funnel-specific page templates we’ve put together along with a few case studies and explanation for how we conducted our research, you can download the complete PDF here.
The templates also include the following:
The case studies these templates draw upon
A walk-through of each template explaining how to use every element
An overview of the methodology behind each template so you can iterate and make them fit your own marketing needs
Please let us know in the comments if you find it helpful or if you see any way to improve it.
But today we’re releasing a new tool to help you get the most value from it – a two-page PDF you can print out and hang in your office. It’s a simple guide for how to use the Conversion Index, with a deeper focus on the most powerful element — motivation. Use it to give you ideas for implementing the Conversion Index in your long-term strategy as well as your day-to-day marketing.
Here is an instant download of the free PDF. Read below for a deeper explanation to help you put this information into action.
(no form to fill out, just click to get your instant download)
Make sure to think about motivation
The Conversion Index is C=4m+3v+2(i-f)-2a. Here is the Conversion Index in a more graphically appealing form:
The “m” stands for “motivation of the user/customer.” It has the highest coefficient — 4 — because motivation has the highest impact on the probability of conversion. Simply put, the better you can tap into a customer’s motivation with your conversion rate optimization, the more likely you are to increase the conversion rate.
Here are some questions to ask to help you optimize for customer motivation:
Who are you optimizing for? You should seek to create a model of your customer’s mind. The more you focus on the customer (versus your product, company, offer, etc.) the more successful you will be. A CEO will need a different landing page than a junior employee. A millennial has different wants and needs than a senior citizen. A risk-averse or cost-conscious buyer will make different decisions than a reward-seeking or risk-taking buyer even when presented with the same information.
Where is your customer in the though sequence? Is this the first time they are interacting with your brand? Or have they had a long relationship with your brand and are ready to buy? Understanding where your customers are in their thought sequence when they get to your landing page — and as they make their way through your landing page — can affect everything on your landing page from the headline to the call-to-action.
Where is the traffic coming from? MarketingSherpa research has shown that organic search has the highest conversion rate, likely because customers are actively looking for something specific when coming from that source. Understanding where your traffic is coming from is another way to understand where they are in the thought sequence. This knowledge can help you provide relevant information to serve their needs and, ultimately, increase conversion.
What conclusions do they need to make? Every product, service, and conversion action has a prospect conclusion funnel. Whether you have mapped it out in detail or not, there are a set of conclusions your customers need to reach before taking any action.
What is the level of urgency?Adding urgency can increase conversion rates. But only if it taps into the customer’s natural motivation. Understand where they might have urgency around the conversion action and use that to optimize your messaging.
What are key pain points and values? What values can you tap into? What pain can your solution help customers overcome? This is key information to ensure your landing page copy and design squarely ties into customer motivations. MarketingSherpa discovered that 23% of marketers consider key pain point an important lead generation form field. This information helps them better meet their prospects’ needs.
What characteristics of your prospect (do you know)? Many marketers tend to focus on demographics and firmographics. But you can’t use that information in a vacuum. For example, you can use prospect characteristics to determine who your best customers will be and treat those prospects different from other prospects.
Once you have the above information, here are some activities to help better understand customers’ motivations and put that understanding into action with your marketing.
Empathize with the customer – Change the way you look at your prospect. They are not simply a target or a lead. Prospects are people just like you. It might seem silly to read that last sentence. Of course, prospects are people. But when we’re relying on databases and technology and driving so hard to meet our goals, it can be easy to overlook the need to empathize with potential customers.
Find someone in that customer type – When you try to create something for the many, it can water down its power. Try to find someone specific in the customer type and write directly to that person. That will help hone your copy. And if possible, talk to them. Get on a call. Meet in person. Try to understand them better. It’s all too easy to assume other people are like us, but they’re not. We are often not the customer, and our goals, fears, vocabulary, patience, drivers and many other characteristics can be very different from the ideal customer type for that product.
Role play with a group – This is a frequent tactic used in everything from boxing to debates to football. Assign roles based on the motivations you’ve discovered about the customer and see how they react to different messaging and offers.
Use personas from Market Intelligence – After analyzing the market, create personas that represent customer motivations. This is a popular marketing tactic. Some marketers like to create specific names for different personas or tie their personas to celebrity examples. Other marketers create personas around different industries or interests. However you create personas, the same general principle applies — writing to a specific customer/customer type will help hone your copy to their motivations.
Now that you have a firm understanding of your customers’ motivations, conduct an analysis of your current and soon-to-launch marketing. Identify gaps between the landing page (and other marketing communications) and the customer’s motivation. These are your opportunities for conversion rate optimization.
Always start with the Conversion Index … but which element?
While motivation is the most impactful element that affects conversion, it doesn’t mean you should necessarily start your CRO right out of the gate by trying to optimize for motivation. There is lower-hanging fruit.
Start with friction and anxiety first, because they are the easiest to see if you put yourself in the customer’s shoes. What can you remove, add or change to reduce these negative elements that hinder conversion?
“If you ignore motivation, it is like you are multiplying by 0 — you are undermining all your other CRO work. If you simply understand and know motivation, it is like you are multiplying by 1 — you aren’t hurting your other CRO work, but you aren’t really helping either.” — Daniel Burstein
Third, leveraging your knowledge of and maximizing for visitor motivation can multiply your business results. This is one reason understanding motivation can be so powerful. You are essentially multiplying with motivation.
“If you ignore motivation, it is like you are multiplying by 0 — you are undermining all your other CRO work. If you simply understand and know motivation, it is like you are multiplying by 1 — you aren’t hurting your other CRO work, but you aren’t really helping either.”
But if you leverage and maximize what you’ve discovered about customer motivation using the previously discussed tactics, it is like you are doubling the impact of all your previous CRO work. Then, you are creating a fluid customer experience that provides value specifically for the reasons your customers want to buy. Not your reasons, but their reasons.If you ignore motivation, it is like you are multiplying by 0 — you are undermining all your other CRO work. If you simply understand and know motivation, it is like you are multiplying by 1 — you aren’t hurting your other CRO work, but you aren’t really helping either.
And that is the most powerful marketing — serving your customers’ motivations.
So, I interviewed Rebecca Strally, Director of Optimization and Design, and Todd Barrow, Director of Application Development, for tips on what considerations you should make for mobile devices when you’re planning and rolling out your tests. Rebecca and Todd are my colleagues here at MECLABS Institute (parent research organization of MarketingExperiments).
Consideration #1: Amount of mobile traffic and conversions
Just because half of global traffic is from mobile devices doesn’t mean half of your site’s traffic is from mobile devices. It could be considerably less. Or more.
Not to mention, traffic is far from the only consideration. “You might get only 30% of traffic from mobile but 60% of conversions, for example. Don’t just look at traffic. Understand the true impact of mobile on your KPIs,” Rebecca said.
Consideration #2: Mobile first when designing responsive
Even if mobile is a minority of your traffic and/or conversions, Rebecca recommends you think mobile first. For two reasons.
First, many companies measure KPIs (key performance indicators) in the aggregate, so underperformance on mobile could torpedo your whole test if you’re not careful. Not because the hypothesis didn’t work, but because you didn’t translate it well for mobile.
Second, it’s easier to go from simpler to more complex with your treatments. And mobile’s smaller form factor necessitates simplicity.
“Desktop is wide and shallow. Mobile is tall and thin. For some treatments, that can really affect how value is communicated.” — Rebecca Strally
“Desktop is wide and shallow. Mobile is tall and thin. For some treatments, that can really affect how value is communicated,” she said.
Rebecca gave an example of a test that was planned on desktop first for a travel website. There were three boxes with value claims, and a wizard below it. On desktop, visitors could quickly see and use the wizard. The boxes offered supporting value.
But on mobile, the responsive design stacked the boxes shifting the wizard far down the page. “We had to go back to the drawing board. We didn’t have to change the hypothesis, but we had to change how it was executed on mobile,” Rebecca said.
Consideration #3: Unique impacts of mobile on what you’re testing
A smartphone isn’t just a smaller computer. It’s an entirely different device that offers different functionality. So, it’s important to consider how that functionality might affect conversions and to keep mobile-specific functionality in mind when designing tests that will be experienced by customers on both platforms — desktop and mobile.
Some examples include:
With the prevalence of digital wallets like Apple Pay and Google Pay, forms and credit card info is more likely to prefill. This could reduce friction in a mobile experience, and make the checkout process quicker. So while some experiences might require more value on desktop to help keep the customer’s momentum moving through the checkout process, including that value on mobile could actually slow down an otherwise friction-lite experience.
To speed load time and save data, customers are more likely to use ad blockers that can block popups and hosted forms. If those popups and forms contain critical information, visitors may assume your site is having a problem and not realize they are blocking this information. This may require clearly providing text explaining about the form or providing an alternative way to get the information, a step that may not be necessary on desktop.
Customers are touching and swiping, not typing and clicking. So information and navigation requests need to be kept simpler and lighter than on desktop.
Visitors can click to call. You may want to test making a phone call a more prominent call to action in mobile, while on desktop that same CTA may induce too much friction and anxiety.
Location services are more commonly used, providing the opportunity to better tap into customer motivation by customizing offers and information in real time and more prominently featuring brick-and-mortar related calls to action, as opposed to desktop, which is in a static location, and the user may be interested in obtaining more information before acting (which may require leaving their current location).
Users are accustomed to app-based experiences, so the functionality of the landing page may be more important on mobile than it is on desktop.
Consideration #4: The device may not be the only thing that’s different
She expanded on that thought, “Do we treat mobile like it is the same audience with the same motivations, expected actions, etc., but just on a different physical device? Or should we be treating those on mobile like a completely different segment/audience of traffic because their motivations, expected actions, etc., are different?”
She gave an example of working with a company her team was performing research services for. On this company’s website, younger people were visiting on mobile while older people were visiting on desktop. “It’s wasn’t just about a phone, it was a different collection of human beings,” she said.
Consideration #5: QA to avoid validity threats
When you’re engaged in conversion optimization testing, don’t overlook the need for quality assurance (QA) testing. If a treatment doesn’t render correctly on a mobile device, it could be that the technical difficulty is causing the change in results, not the changes you made to the treatment. If you are unaware of this, it will mislead you about the effectiveness of your changes.
Here are some of the devices our developers use for QAing.
(side note: That isn’t a stock photo. It’s an actual picture by Senior Designer James White. When I said it looked too much like a stock image, Associate Director of Design Lauren Leonard suggested I let the readers know “we let the designers get involved, and they got super excited about it.”)
“If your audience are heavy users of Safari on iPhone, then check on the actual device. Don’t rely on an emulator.” — Todd Barrow
“Know your audience. If your audience are heavy users of Safari on iPhone, then check on the actual device. Don’t rely on an emulator. It’s rare, but depending on what you’re doing, there are things that won’t show up as a problem in an emulator. Understand what your traffic uses and QA your mobile landing pages on the actual physical devices for the top 80%,” Todd advised.
Consideration #6: The customer’s mindset
Customers may go to the same exact landing page with a very different intent when they’re coming from mobile. For example, Rebecca recounted an experiment with an auto repair chain. For store location pages, desktop visitors tended to look for coupons or more info on services. But mobile visitors just wanted to make a quick call.
“Where is the customer in the thought sequence? Mobile can do better with instant gratification campaigns related to brick-and-mortar products and services,” she said.
Consideration #7: Screen sizes and devices are not the same things
Most analytics platforms give you an opportunity to monitor your metrics based on device types, like desktop, mobile and tablet. They likely also give you the opportunity to get metrics on screen resolutions (like 1366×768 or 1920×1080).
Just keep in mind, people aren’t always viewing your websites at the size of their screen. You only know the size of the monitor not the size of the browser.
“The user could be recorded as a full-size desktop resolution, but only be viewing in a shrunken window, which may be shrunk down enough to see the tablet experience or even phone experience,” Todd said. “Bottom line is you can’t assume the screen resolutions reported in the analytics platform is actually what they were viewing the page at.”
Consideration #8: Make sure your tracking is set up correctly
Mobile can present a few unique challenges for tracking your results through your analytics and testing platforms. So make sure your tracking is set up correctly before you launch the test.
For example, if you’re using a tag control manager and tagging things through it based on CSS properties, if the page shifts at different breakpoints that change the page structure, you could have an issue.
“If you’re tagging a button based on its page location at the bottom right, but then it gets relocated on mobile, make sure you’re accounting for that,” Todd advised.
Also, understand how the data is being communicated. “Because Google Tag Manager and Google Optimize are asynchronous, you can get mismatched data if you don’t follow the best practices,” Todd said.
“If you see in your data that the control has twice as many hits as the treatment, there is a high probability you’ve implemented something in a way that didn’t account for the way asynchronous tags work.” —Todd Barrow
Todd provided a hard-coded page view as an example. “Something to watch for when doing redirect testing … a tracking pixel could fire before the page loads and does the split. If you see in your data that the control has twice as many hits as the treatment, there is a high probability you’ve implemented something in a way that didn’t account for the way asynchronous tags work. This is really common,” Todd said.
“If you know that’s going to happen, you can segment the data to clean it,” he said.
Real-world behavioral tests are an effective way to better understand your customers and optimize your conversion rates. But for this testing to be effective, you must make sure it is accurately measuring customer behavior.
One reason these A/B split tests fail to give a correct representation of customer behavior is because of validity threats. This series of checklists is designed to help you overcome Instrumentation Effect. It is based on actual processes used by MECLABS Institute’s designers, developers and analysts when conducting our research services to help companies improve marketing performance.
MECLABS defines Instrumentation Effect as “the effect on the test variable caused by a variable external to an experiment, which is associated with a change in the measurement instrument.” In other words, the results you see do not come from the change you made (say, a different headline or layout), but rather, because some of your technology has affected the results (slowed load time, miscounted analytics, etc.)
Avoiding Instrumentation Effect is even more challenging for any test that will have traffic from mobile devices (which today is almost every test). So, to help you avoid the Instrumentation Effect validity threat, we’re providing the following QA checklist. This is not meant for you to follow verbatim, but to serve as a good jumping-off point to make sure your mobile tests are technically sound. For example, other browsers than the ones listed here may be more important for your site’s mobile functionality. Maybe your landing page doesn’t have a form, or you may use different testing tools, etc.
Of course, effective mobile tests require much more than thorough QA — you also must know what to test to improve results. If you’re looking for ideas for your tests that include mobile traffic, you can register for the free Mobile Conversion micro course from MECLABS Institute based on 25 years of conversion optimization research (with increasing emphases on mobile traffic in the last half decade or so).
There’s a lot of information here, and different people will want to save this checklist in different ways. You can scroll through the article you’re on to see the key steps of the checklist. Or use the form on this page to download a PDF of the checklist.
The following checklists are broken out by teams serving specific roles in the overall mobile development and A/B testing process. The checklists are designed to help cross-functional teams, with the benefit being that multiple people in multiple roles bring their own viewpoint and expertise to the project and evaluate whether the mobile landing page and A/B testing are functioning properly before launch and once it is live.
For this reason, if you have people serving multiple roles (or you’re a solopreneur and do all the work yourself), these checklists may be repetitive for you.
Here is a quick look at each team’s overall function in the mobile landing page testing process, along with the unique value it brings to QA:
Dev Team – These are the people who build your software and websites, which could include both front-end development and back-end development. They use web development skills to create websites, landing pages and web applications.
For many companies, quality assurance (QA) would fall in this department as well, with the QA team completing technical and web testing. While a technical QA person is an important member of the team for ensuring you run valid mobile tests, we have included other functional areas in this QA checklist because different viewpoints from different departments will help decrease the likelihood of error. Each department has its own unique expertise and is more likely to notice specific types of errors.
Value in QA: The developers and technological people are most likely to notice any errors in the code or scripts and make sure that the code is compatible with all necessary devices.
Project Team – Depending on the size of the organization, this may be a dedicated project management team, a single IT or business project manager, or a passionate marketing manager keeping track of and pushing to get everything done.
It is the person or team in your organization that coordinates work and manages timelines across multiple teams, ensures project work is progressing as planned and that project objectives are being met.
Value in QA: In addition to making sure the QA doesn’t take the project off track and threaten the launch dates of the mobile landing page test, the project team are the people most likely to notice when business requirements are not being met.
Data Team – The data scientist(s), analyst(s) or statistician(s) helped establish the measure of success (KPI – key performance indicator) and will monitor the results for the test. They will segment and gather the data in the analytics platform and assemble the report explaining the test results after they have been analyzed and interpreted.
Value in QA: They are the people most likely to notice any tracking issues from the mobile landing page not reporting events and results correctly to the analytics platform.
Design Team – The data scientist(s), analyst(s) or statistician(s) helped establish the measure of success (KPI – key performance indicator) and will monitor the results for the test. They will segment and gather the data in the analytics platform and assemble the report explaining the test results after they have been analyzed and interpreted.
Value in QA: They are the people most likely to notice any tracking issues from the mobile landing page not reporting events and results correctly to the analytics platform.
DEV QA CHECKLIST
Pre-launch, both initial QA and on Production where applicable
Visual Inspection and Conformity to Design of Page Details
Verify latest copy in place
Preliminary checks in a “reference browser” to verify design matches latest comp for desktop/tablet/mobile layouts
Use the Pixel Perfect Overlay function in Firefox Developer Tools – The purpose of this tool is to take an image that was provided by the designer and lay it over the website that was produced by the developer. The image is a transparency which you can use to point out any differences or missing elements between the design images and the webpage.
Displaying of images – Make sure that all images are displaying, aligned and up to spec with the design.
Forms, List and Input Elements (Radio Buttons, Click Boxes) – Radio buttons (Dots and Circles) and Checkboxes (Checks and Boxes) are to be tested thoroughly as they may trigger secondary actions. For example, selecting a “Pay by Mail” radio button will sometimes automatically hide the credit card form.
Margins and Borders – Many times, you will notice that a portion of the body or perhaps a customer review or image is surrounded by a border or maybe even the whole page. It is our duty to inspect them so that there are no breaks and that they’re prominent enough for the user to decipher each bordered section.
Copy accuracy – Consistency between typography, capitalization, punctuation, quotations, hyphens, dashes, etc. The copy noted in the webpage should match any documents provided pertaining to copy and text unless otherwise noted or verified by the project manager/project sponsor.
Font styling (Font Color, Format, Style and Size) – To ensure consistency with design, make sure to apply the basic rules of hierarchy for headers across different text modules such as titles, headers, body paragraphs and legal copies.
Link(s) (Color, Underline, Clickable)
Web Page Functionality: Verify all page functionality works as expected (ensure treatment changes didn’t impact page functionality)
Top navigation functionality – Top menu, side menu, breadcrumb, anchor(s)
Links and redirects are correct
Media – Video, images, slideshow, PDF, audio
Form input elements – drop down, text fields, check and radio module, fancy/modal box
Form validation – Error notification, client-side errors, server-side errors, action upon form completion (submission confirmation), SQL injection
Verify split functional per targeting requirements
Verify key conversion scenario (e.g., complete a test order, send test email from email system, etc.) – If not already clear, QA lead should verify with project team how test orders should be placed
Where possible, visit the page as a user would to ensure targeting parameters are working properly (e.g., use URL from the PPC ad or email, search result, etc.)
Verify tracking metrics are firing in browser, and metric names match requirements – Check de-bugger to see firing as expected
Verify reporting within the test/analytics tool where possible – Success metrics and click tracking in Adobe Target, Google Content Experiments, Google Analytics, Optimizely, Floodlight analytics, email data collection, etc.
Back End Admin Panel
Database includes accurate offer information
Tracking correct order information from customer (data collection)
Payment Validation and Errors – Ensure the character limiter is effective for all types of credit cards (Visa & Mastercard: 16, Amex: 14)
Notify Project Team and Data Team it is ready for their QA (via email preferably) – indicate what reference browser is. After Project Team initial review, complete full cross browser/ cross device checks using “reference browser” as a guide:
Browser Functionality – Windows
Internet Explorer 7 (IE7)
Browser Functionality – macOS
Mobile Functionality – Tablet
Mobile Functionality – Mobile phone
Post-launch, after the test is live to the public:
Notify Project Team & Data Team the test is live and ready for post-launch review (via email preferably)
Verify split is open to public Verify split functional per targeting requirements
Where possible, visit the page as a user would to ensure targeting parameters are working properly (e.g., use URL from the PPC ad or email, search result, etc.)
Test invalid credit cards on a production environment
PROJECT TEAM QA CHECKLIST:
Pre-Launch and Post-Launch QA:
Check that copy and design are correct for control and treatments in the “reference browser”:
Ensure all added copy/design elements are there and correct
Ensure all removed copy/design elements are gone
Ensure all changed copy/design elements are correct
Ensure control experience is as intended for the test
Check page functionality:
Ensure all added/changed functionality is working as expected
Ensure all standard/business as usual – BAU_ functionality is working as expected:
Go through the typical visitor path (even beyond the testing page/ location) and ensure everything functions as expected
Make sure links go where supposed to, fields work as expected, data passes as expected from page to page.
Check across multiple browser sizes (desktop, tablet, mobile)
If site is responsive, scale the browser from full screen down to mobile and check to ensure all the page breaks look correct
Where possible, visit the page the way a typical visitor would hit the page (e.g., through PPC Ad, organic search result, specific link/button on site, through email)
DATA QA CHECKLIST:
Pre-Launch QA Checklist (complete on Staging and Production as applicable):
Verify all metrics listed in the experiment design are present in analytics portal
Verify all new tracking metrics’ names match metrics’ names from tracking document
Verify all metrics are present in control and treatment(s) (where applicable)
Verify conversion(s) are present in control and treatment(s) (where possible)
Verify any metrics tracked in a secondary analytics portal (where applicable)
Immediately communicate any issues that arise to the dev lead and project team
Notify dev lead and project team when Data QA is complete (e-mail preferably)
Post-Launch QA / First Data Pull:
Ensure all metrics for control and treatment(s) are receiving traffic
Ensure traffic levels are in line with the pre-test levels used for test duration estimation
Update Test Duration Estimation if necessary
Immediately communicate any issues that arise to the project team
Notify dev lead and project team when first data pull is complete (e-mail preferably)