Stop Paying Attention to Split Test Results

Video Transcript

The other day I’m writing a blog post and I go to Slack’s home page real quick. And you probably know what Slack is by now, it’s the super popular messaging app that like everyone and their mom is using right now. So anyway I get to the home page and I’ve been there a few days before and I notice that this time things look a little bit different than they had in the past. And by a little bit different I mean they look completely different. So what I figured is they must be running a split test. So I open up a window in a new browser, refresh the page a few times and sure enough the version that I remember previously being there comes up. And so I’ve got the new version of Slack’s home page in one window and version I previously remember being there in the other.

So they’re running a split test and I immediately think to myself, OH MY GOD THIS IS AWESOME. It would be so cool to see the results of this. I even got all excited. I made a little tweet about it. I like took screenshots of each home page and cropped them together and tweeted it. And was super excited about it and then I stopped. And I was looking at the test little bit more and I thought to myself…why? Why would I be interested in the results of this split test? If one version got a higher conversion rate than the other would I just immediately take sort of the principles that are at work in that winning variation and apply them to my own home page?

I probably wouldn’t. But I might think it’s a good test to run on my own home page, and get a test together between the principles at work in what Slack is testing and what I’ve been doing on my own page. So really the conclusion I was left with was that I already have the best information. If I’m going to test anyway, then the real value in stumbling upon the split test isn’t the result. It’s the thought process behind what Slack is testing.

So I think a lot of times people see results that someone has gotten with a split test and they just immediately think, “Hey you know this button color won for them or this type of headline won for them, I bet it’ll work for me,” and they just immediately apply it without thinking about the differences between their own business and the business that test was run in. I’m guilty of this even. I’m looking at Slack’s home page, which is a company valued at over a billion dollars. They have millions of users. Probably the fastest growing company, or one of the fastest growing companies in the world right now. And I’m just thinking, “Yeah sure whatever works for Slack, that’ll work for me. That will work for Copygrad, my home page where I sell copywriting services.”

We run into this all the time and so I want to show you these two split tests. And take you through what I think they’re they’re trying to test and determine about their brand and what converts best for it right now. And how you can take the same things that they’re testing just by looking at the test and thinking about it for a few minutes, and you can run the same kind of tests in your own business. Intelligently. And actually apply what they’re doing and sort of formulate it so that it makes sense for your business so you can test it. Rather than just seeing a result and immediately clinging onto it and thinking it’s going to work for you. So let’s take a look at those two homepage variations right now.

So the control page is really all about aspiration. Slack’s basically saying this is a tool that companies doing amazing things are using, and they use an example from NASA right here with you know putting a robot on Mars and they’re very enthusiastic with underlining and exclamation points.

So this is really no different than what you see in something like a Nike commercial where you have pro athletes running around doing awesome stuff in their shoes. As consumers, we’ve always had a strong desire to use the same tools and products that the pros are using. So this version right here really really plays on that angle.

The variation that, even though it’s using the same value proposition right here, even though that value proposition stays the same, it’s framed in a truly different way. So the angle here is all about measurable results. It’s data driven and concrete. So you have kind of four different sections that it goes through. So it’s all of this very specific data around specific pain points. Again, much different than what the control does in being more aspirational and right-brained.  And there’s a reason Slack would run a test like this. And that’s because they know both versions of the page utilize proven, extremely powerful motivators. They’re most likely just trying to figure out right now which better motivates their prospects. Both obviously have merit and work. They’re just trying to see what works for them right now.

But the real value is that you can now take the principles behind this test and apply them to any other business. So I’m going to take just a random web site in a totally different industry. And let’s let’s just look at how this would be applied. So I’m going to do this with Red Wings shoes. Because I bought a pair of these boots when I lived in Minnesota, which is where they’re made and they’re awesome.

And you can see that right now they’re all about craftsmanship on this page. So they’re really emphasizing this as a handmade, premium, beautiful product which is an angle you see working in a lot of different industries right now.

So let’s start by giving this a more aspirational angle. And if you scroll down you actually see they have this Faces of red wing section where they have this guy who runs the oldest distillery in New York City. So he’s this rugged dude surrounded by I think whiskey barrels, and he’s wearing their heritage booths. And it’s just like who wouldn’t want to live this guy’s life.

So if they want to experiment with an aspirational angle they could take someone like this guy and kind of put him front and front and Center and make the copy about how people doing cool rugged manly stuff like owning the oldest whiskey distillery in New York wear Red Wing boots. And then to take the other approach of being more data driven and performance based, they could make it all about something like how long these boots will last and kind of tie it back into the heritage angle or something like, “Boots that last so long they will stay in your family forever. Or something like that or maybe they know on average how long a customer goes in between purchasing a replacement pair, maybe that’s like a decade or something like that.

So that would be an easy way for a company like Red Wing in a totally different industry to experiment with the principles at work in Slack’s test.

So what do we usually say when someone is telling a story about how they accomplished some amazing, unbelievable result? The person usually says something to the effect of a really popular saying: The journey is the reward.

It wasn’t the end result, it was the process of getting there that was really the rewarding part when I look back now. And that’s how I look at split tests run by other people where I don’t have access to the results and everything. It’s not the result that is going to be the rewarding part. The rewarding part is their journey. So their journey is your reward.

You can compare the variations and you’ll get a sense of what companies are doing, what they’re trying to find out about what their customers respond to, what they’re trying to find out about what’s appealing about their product. There’s so much more. And so just really look for that from now on when you see split tests.