Anyone who works in or follows the enterprise content management (ECM) industry will remember when Gartner redefined ECM and changed the term to content services. Michael Woodbridge, a research director at Gartner, outlined the reasons for the change in a blog post at the beginning of 2017, and ECM hasContinue reading...
Recent research from Gartner underscores something many digital workers have suspected for quite some time. At least 84 percent of consumers across all industries say their experiences using digital tools and services fall short of expectations.Continue reading...
Facebook isn't the only one with a measurement problem (though it may well be the most publicized.) The problem exists across apps everywhere — and before you jump to conclusions, remember this. Siloed measurement practices are not intentionally designed to mask the truth or hold back information for nefarious reasons.Continue reading...
We talk a lot about a/b testing here on MarketingExperiments. What we don’t usually talk about is a/b testing for the mobile web…especially testing within mobile apps.
I thought we should change that. As I was scouring the web looking for mobile a/b tests, I found this 2-year old video by Amazon.
Apparently, Amazon Web Services (AWS) at one point had an a/b testing feature that is now closed
When they had testing, however, one app developer used it extensively and shared their experiences in a promotional video for the feature on Amazon. The developers were behind the game Air Patriots. Russell Caroll was the Senior Producer for the game and Julio Gorge was the Game Development Engineer. The game is a kind of aerial take on the classic tower defense game genre.
Now granted, this was a promotional video, but the content still speaks for itself. These guys had (and still have by the looks of it) a fairly successful mobile app and they ran some successful tests. It’s a great starting place for what you can test in your mobile app.
By the way, while Amazon has shut down its a/b testing feature, there are a lot of other tools for testing mobile apps that will accomplish the same thing the developers talk about in the video.
Test #1: What is the impact of ads on customer experience? (1:34)
The first thing the team tested was the impact of ads are on their customers. They wanted to make sure the ads did not harm the customer experience. So they tested a single ad in the main menu near the bottom of the screen.
They found that the ads didn’t affect customer retention. This meant that they could insert ads and generate more revenue without hurting their customers.
Test #2: Will in-game ad placement affect customer retention? (2:56)
In the second test, the team put ads in the game screen.
In both the first and second tests, the ads had a little “X” that the customers could tap to hypothetically dismiss the ads. When they tapped, a pop up came up that told customers they could eliminate ads with any purchase in the game’s store.
In this test, there was again, no impact on customer retention, but there was a statistically significant increase in revenue.
Test #3: Simple game-circle icon test (4:20)
In this test, the team wanted to know whether an icon to the game-circle (Amazon’s game stats and leaderboards portal) would improve performance.
It’s not clear which icon won, or even why this particular test was useful for the team, but they did get a favorable result, and the lesson they wanted to drive home was that simple changes like icons can make a difference. We’ve, of course found that to be the case in a large number of our tests on MarketingExperiments.
Test #4: Does game difficulty affect revenue? (4:58)
In this 4th test, Caroll made a mistake. He accidentally changed the game difficulty to make it about 10% harder. As a result, every metric that was important to them tanked.
The team of course fixed it as fast as possible, but it gave them an idea.
What would happen to revenue if they made the game easier?
So they ran a test that had 5 treatments: The control and then 4 difficulty levels that were easier than the control.
It turned out that the easiest difficulty performed the best. By making it easier, they players playing 20% longer and revenue went up 20%.
Test #5: When is the best time to have push-notifications for re-engagement in inactive players? (7:43)
The team then tested a push-notification that offered inactive players and incentive for picking the game back up.
They wanted to know when the best time to send the notification would be. So they tested a few different variables and found that the best time was 3 days after the last game play.
They also found that sending the notification 7 days after the last game play negatively impacted their performance metrics.
With these 5 tests and probably a few more that have been happening off the record, the team was able to develop a great app for their customers and steadily increase their revenue. At the end of the video Carol gives a few key takeaways for marketers who are a/b testing their apps.
You might also like:
SAN FRANCISCO — Do you really need yet another app for your phone to remind you about work and all those tasks you need to accomplish? Salesforce thinks you do.Continue reading...
At its Worldwide Developers Conference in June, Apple announced changes coming to iOS 10 (now available in beta) that will make push notifications more powerful than ever.Continue reading...
Tel Avi-based SimilarWeb has incorporated the mobile intelligence functionality it acquired from Quettra last year into a new offering called SimilarWeb Portrait. Portrait's mission is straightforward: to track the app’s user behavior, which is anonymized, from the moment the app is downloaded. Armed with this user data, or portraitContinue reading...
While the vast majority of mobile devices worldwide run Google’s Android operating system, Apple’s App Store that remains the cash cow. This is the case even though Google Play downloads surpassed those from iPhones and iPads in the United States for the first time.Continue reading...
Almost 9 percent of popular apps downloaded from Google Play interact with websites that could compromise users’ security and privacy, according to a study released this month by researchers at the University of California, Riverside.Continue reading...