5 CRO Questions Answered – Optimization Process, Testing & Troubleshooting



Since I started CRO at Shopify, I’ve been asked a lot of questions about e-commerce and marketing optimization. Most of these questions are centred around website testing, and to be perfectly honest, I had the same questions when I started too!

So without further ado, here are the top questions I’m asked as a CRO answered:

Where do I start? 

Trying to figure out where to start is one of the biggest hurdles. When I am looking for testing and conversion rate optimization opportunities I look at two major areas:

Pages that cost the most money: Spending your advertising dollars wisely is crucial when you’re running a company. Whether your spending money on Google, Bing or Facebook you want to make sure that every dollar counts. Bryan Eisenberg once wrote that companies typically spend $92 to bring customers to their site, and spend around $1 to convert them. By focusing on where you spend the most money you have the greatest chance of getting more bang for your buck (that’s ROI for you business types.)

Pages with the most traffic: Don’t forget about your other sources! While it’s important to look at the pages that cost you the most and bring the most paid traffic to your website, don’t dismiss the pages that might be getting organic traction. These are great places to focus on conversion and best of all, it’s free traffic!

Beyond that, try to focus on the most dramatic tests first. Test completely different layouts or major ideas first. This is so you won’t get stuck working on tests that might only have small results (10% or less). Don’t worry if you fail big on these tests, the best thing about dramatic tests like these is that they can lead to big learnings. This will help you figure out what your customers need faster.

How should I structure my test?

There are a lot of amazing articles out there that outline different ways to structure your tests. Some that come to mind are Peep Laja’s from ConversionXL and Chris Goward’s LIFT model from Wider Funnel.

When I structure tests i try to make it as easy as possible:

  1. Create a blue sky goal: Think of yourself at a Roulette table in Vegas with a stack of free money. The bigger the bet, the bigger the learnings. (i.e.: Removing the Nav on my landing page will increase visit to signup conversion rate by 100%.) Don’t be afraid if you’re off, you’ll get better at this over time but this helps you get into a “blue sky” mindset. Focusing on tests that will give you big results will get you faster to big wins.
  2. Set up your test: Whether you’re running an A/B test or a multi-variate test using unbounce or testing in-house make sure you set up your test properly. Are you targeting a particular source? Has your landing pages been checked for errors?
  3. Decide how confident you need to be for your test: Do you need 95% confidence? Do you need x number of conversions? Whatever it is that you need to validate your test, decide beforehand. Get stakeholders involved in this so you don’t need to answer to them after the test is already done.
  4. Keep Track of Results & Learnings: Results are great but I would make sure you keep track of what you learned from the experient. Was it a waste of time? Should you have focused on a completely different segment? What tests would you want to do later as a result of this test? These are all important findings and should be tracked and shared with your team.

How long should i run my test?

In the perfect scenario..

When I run a test I want to be confident that the results are what I’m looking for. I usually use a tool like Avinash Kaushik’s Significance Calculator or Kissmetrics Calculator to decide whether or not my test is at least 95% confident. That is the perfect scenario.

Sometimes things aren’t so perfect…

In some situations you either have a deadline or you need to show some results fast. Here’s what I would do. Start your test and don’t look at it for at least a week. After the first week take note of what you see. Are there very clear results? Is the results still inconclusive or have minimal results? (under 10%) Start thinking of what you’ve learned from this and what questions you need answered in the next week. At week two if one your variations are doing much better than the other, replace the original. Spend the next week or two seeing if these results really make a difference conversion wise.

Remember, your test is only telling you that in MOST cases one variation will do better than the other, you still need to see if it’s true. Take a week or two and see what happens. In the meantime you can track and share your findings and start working on your next move.

Another pro tip: remember to start and end your test on the same day. In most cases you have consistent days where things are higher/lower than normal, you want to take this into account when you’re testing.

What happens if i see nothing after 2 weeks?

At week two it’s time to make a decision either way. If your results are still inconclusive and are showing minimal results, I would scrap it. You can do better than that. Since you should have already written down your initial findings a week ago, you probably already have some new areas to test.

What do I do after the test?

Keep testing, of course! Take what you learned from your test, share your findings and implement. Keep an eye on the landing page to make sure that it gets the conversion increase you were hoping for and start testing another area of interest.

The most important thing to remember is…

Don’t get lost in the details – start testing now!  

It’s easy to get overwhelmed by all the numbers and lose steam when a test doesn’t go your way. Don’t be discouraged. Think of it this way, whether or not the results you encounter after testing leads to an increase in conversion or not it will ultimately get you closer to understanding your customers needs. This will get you closer to giving them what they want. Isn’t that what you wanted all along?

More Questions or Comments?

If you have any other questions, please feel free to post them in the comments below and I’d be happy to answer them. I also encourage anyone else who has been testing to give their answers to the following questions as well. The more the merrier!

Questions? Comments?