5 CRO Questions Answered – Optimization Process, Testing & Troubleshooting



Since I started CRO at Shopify, I’ve been asked a lot of questions about e-commerce and marketing optimization. Most of these questions are centred around website testing, and to be perfectly honest, I had the same questions when I started too!

So without further ado, here are the top questions I’m asked as a CRO answered:

Where do I start? 

Trying to figure out where to start is one of the biggest hurdles. When I am looking for testing and conversion rate optimization opportunities I look at two major areas:

Pages that cost the most money: Spending your advertising dollars wisely is crucial when you’re running a company. Whether your spending money on Google, Bing or Facebook you want to make sure that every dollar counts. Bryan Eisenberg once wrote that companies typically spend $92 to bring customers to their site, and spend around $1 to convert them. By focusing on where you spend the most money you have the greatest chance of getting more bang for your buck (that’s ROI for you business types.)

Pages with the most traffic: Don’t forget about your other sources! While it’s important to look at the pages that cost you the most and bring the most paid traffic to your website, don’t dismiss the pages that might be getting organic traction. These are great places to focus on conversion and best of all, it’s free traffic!

Beyond that, try to focus on the most dramatic tests first. Test completely different layouts or major ideas first. This is so you won’t get stuck working on tests that might only have small results (10% or less). Don’t worry if you fail big on these tests, the best thing about dramatic tests like these is that they can lead to big learnings. This will help you figure out what your customers need faster.

How should I structure my test?

There are a lot of amazing articles out there that outline different ways to structure your tests. Some that come to mind are Peep Laja’s from ConversionXL and Chris Goward’s LIFT model from Wider Funnel.

When I structure tests i try to make it as easy as possible:

  1. Create a blue sky goal: Think of yourself at a Roulette table in Vegas with a stack of free money. The bigger the bet, the bigger the learnings. (i.e.: Removing the Nav on my landing page will increase visit to signup conversion rate by 100%.) Don’t be afraid if you’re off, you’ll get better at this over time but this helps you get into a “blue sky” mindset. Focusing on tests that will give you big results will get you faster to big wins.
  2. Set up your test: Whether you’re running an A/B test or a multi-variate test using unbounce or testing in-house make sure you set up your test properly. Are you targeting a particular source? Has your landing pages been checked for errors?
  3. Decide how confident you need to be for your test: Do you need 95% confidence? Do you need x number of conversions? Whatever it is that you need to validate your test, decide beforehand. Get stakeholders involved in this so you don’t need to answer to them after the test is already done.
  4. Keep Track of Results & Learnings: Results are great but I would make sure you keep track of what you learned from the experient. Was it a waste of time? Should you have focused on a completely different segment? What tests would you want to do later as a result of this test? These are all important findings and should be tracked and shared with your team.

How long should i run my test?

In the perfect scenario..

When I run a test I want to be confident that the results are what I’m looking for. I usually use a tool like Avinash Kaushik’s Significance Calculator or Kissmetrics Calculator to decide whether or not my test is at least 95% confident. That is the perfect scenario.

Sometimes things aren’t so perfect…

In some situations you either have a deadline or you need to show some results fast. Here’s what I would do. Start your test and don’t look at it for at least a week. After the first week take note of what you see. Are there very clear results? Is the results still inconclusive or have minimal results? (under 10%) Start thinking of what you’ve learned from this and what questions you need answered in the next week. At week two if one your variations are doing much better than the other, replace the original. Spend the next week or two seeing if these results really make a difference conversion wise.

Remember, your test is only telling you that in MOST cases one variation will do better than the other, you still need to see if it’s true. Take a week or two and see what happens. In the meantime you can track and share your findings and start working on your next move.

Another pro tip: remember to start and end your test on the same day. In most cases you have consistent days where things are higher/lower than normal, you want to take this into account when you’re testing.

What happens if i see nothing after 2 weeks?

At week two it’s time to make a decision either way. If your results are still inconclusive and are showing minimal results, I would scrap it. You can do better than that. Since you should have already written down your initial findings a week ago, you probably already have some new areas to test.

What do I do after the test?

Keep testing, of course! Take what you learned from your test, share your findings and implement. Keep an eye on the landing page to make sure that it gets the conversion increase you were hoping for and start testing another area of interest.

The most important thing to remember is…

Don’t get lost in the details – start testing now!  

It’s easy to get overwhelmed by all the numbers and lose steam when a test doesn’t go your way. Don’t be discouraged. Think of it this way, whether or not the results you encounter after testing leads to an increase in conversion or not it will ultimately get you closer to understanding your customers needs. This will get you closer to giving them what they want. Isn’t that what you wanted all along?

More Questions or Comments?

If you have any other questions, please feel free to post them in the comments below and I’d be happy to answer them. I also encourage anyone else who has been testing to give their answers to the following questions as well. The more the merrier!

How to Raise Conversion Rates 100% Like a Super Hero in a Fast Growth Company (Even when you don’t feel like one)

“Give me an idea that’s going to raise conversion rates AT LEAST 100%”

Well that’s daunting, but not very far from the truth of a conversion rate optimizer in a high growth startup.

When I face a conversation like this, I actually wish I was new to the industry, to not have done so many things, or worked on so many websites.

Then I would be able to confidently look someone in the eye, all rosy faced ready to take on the world and shout,

“I’m on it.”

But unfortunately that’s not me. While I’ve worked on over 300 websites and done everything from SEO, SEM, Social Media & even content writing for a celebrity blog I hear 100% conversion rate increase and one thing happens to me.

I become absolutely paralyzed.

I wish I could say I was like the heroes of the online marketing world who write the blog articles we read everyday who seem to laugh in the face of these obstacles. I wish, especially as a tech women to be able to eat 100% conversion rate increases for breakfast.

But alas, I’m not a super hero. I’m just some girl who really freakin’ loves online marketing and I really love a challenge.

Challenge accepted

So where do I start?

Your instinct after a meeting like this will be to do one or all of the following:

  1. Run to your desk
  2. Start ordering around developers, designers & online marketing to do something NOW..
  3. Realize you have no idea what they should be working on.
  4. Huff and puff about how you’ll never think of anything.
  5. Panic
  6. Flip a table and run


So let’s throw out all of the above and do this instead:

  1. Walk calmly to your desk
  2. Take a couple of breaths
  3. Grab a notebook
  4. Start idea hustling

What’s the idea hustle?

The idea hustle is where you look at everything you have done, everything you thought you should do, and what everyone wishes you would do.

Your goal: Create a list of ideas that you can do today, tomorrow, or later that will: 1) solve problems 2) can be done quickly 3) get the results you need.

What can be done today?

Up until this point you’ve probably done quite a few tests. You’ve probably learned some things like which style of landing page works, what social proof is more effective, what content resonates with your audience. While you haven’t tested EVERY page you’ve probably seen some trends that have worked well for certain keywords.

So what can you do with this information:

  1. Write down everything you’ve learned from this point.
  2. Bring your team together and ask them where these rules can be applied to make the biggest impact. (i.e.: landing pages for adword groups or high trafficked SEO pages)
  3. Ask your team members (hopefully composed of designers & developers) how this can be streamlined, templated, and pushed out efficiently.
  4. Make it happen.
  5. Keep an eye on it.

Now I know what you’re thinking, “But I have to test… EVERYTHING!”

While this is true, you also have to provide value and drive results. Hiding behind tests is the risk adverse way of doing things (and trust me I’m probably as risk adverse as they come) but in order to see the benefits of your work you need to trust what you’ve learned up until this point and test it on a grand scale.

My only advice? Take #5 seriously. While some areas will give you a huge lift, there will be some pages that don’t and you want to revert back quickly.

What can be done tomorrow?

Now, I’m using the term “tomorrow” loosely but the implication is thinking about things that can be tested quickly.

This requires research. While it would be great to be the one person who thinks of that huge 100% conversion rate lift all on their very own  it doesn’t always happen that way. Your job is to get things done and show results.

This happens by asking a lot of questions to a lot of people. Another tip? Don’t get stuck in an endless stream of meetings. Your goal is to do this quickly and efficiently so it may mean hallway chats on the way to the coffee machine.

Who do you talk to?

Grab that notebook and start interviewing. You should probably start with this list:

  • Your customers – They will ultimately be the ones helping you reach your goal right? Read bug tickets, listen in on customer service calls, read work your UX team has done (if you have one), add a qualaroo survey to your site and read what they have to say or send out a survey. Find out what they love about your product, what’s stopping them from using your product and what they need from you.
  • Your Direct manager -  try to understand his or her perspective of what needs to be done, what advice they have to offer, and what areas seem to be in the most trouble. (Don’t have one? See #2)
  • The HIPPO (Highest Paid Person Opinion) – Whether it’s a senior executive, the CEO or an investor. Learn what makes them tick, where they think the problem is and the areas they look at. This will help you get outside of your world and get a broader top down perspective of the areas that affect their day to day most.
  • Talk to your team mates - These are your designers, developers, online marketers, and growth hackers. They ALL have different areas they find problematic and advice to give.

What do you do with all that information?

Chris Goward, founder and CEO of Wider Funnel and author of “You Should Test That” introduced an excellent way to prioritize conversion rate optimization tests called The PIE Framework. It breaks up ideas by Potential, Importance, and Ease.

Organizing your research into PIE

PIE Framework

PIE Framework: Chris Goward, Wider Funnel

Potential: This is where you would take traffic and size of audience into consideration. Does this test affect most of your visitors and prospects? Will a 100% conversion rate make the biggest impact here?

Importance: This is where your interviews become important. Is this an area your customers or HIPPOS found important? Rank these higher than the others.

Ease: Your designers, developers, and online marketers will give you the answer to these. How easy are these changes? How long will it take? If it can’t be done in 1 day – 1 week, then it should be low on your list. You’re in a high growth fast moving company remember?

Now you have a list of things that can be done “tomorrow” and things that can be done later.

With that we go back to what can be done today:

  1. Take your PIE Framework
  2. Bring your team together and show them what you’ve learned from all your interviewing and research and why you chose these high priority items.
  3. Ask your team members how this can be streamlined, templated, and pushed out efficiently.
  4. Make it happen.
  5. Keep an eye on it.

What can be done later?

Who cares? You’re in a fast growth company. Later doesn’t matter.

“Give me an idea that’s going to raise conversion rates AT LEAST 100%”

Your new answer: “How about I show you?”

A statement like this doesn’t have to be a paralyzing event if you take a breath, speak to the right people, reiterate fast and efficiently and show some results.

#TLE2014 Cheat Sheet – 6 Ways to Convert Using Persuasion

As promised, here’s the #TLE2014 Cheat sheet** for my talk on 6 Ways to Use Persuasion to Convert!

Cialdini’s 6 Principles of Persuasion

In 1984 Robert Cialdini wrote the book Influence. In the book he listed off six persuasion principles.

In the next few sections you will learn a bit more about each of the 6 persuasion techniques along with examples, and additional resources for even more fun stuff!


Definition: This is the idea that when someone gives you something, you feel obligated to return the favor.


1) Moving your friends

Moving Your Friends - Reciprocity

2) Free Trials

Free Trial - Reciprocity

Read more: What Goes Around Comes Around: Reciprocity on the web

Consistency & Commitment

The consistency & commitment principle, according to Cialdini asserts that people are more likely to honor a commitment if it matches their self image.


1) Udemy

Udemy - Commitment

If you choose a Boss they will send you regular check-ins answering two questions:

  • What have you accomplished?
  • What do you aim to accomplish and by when?

2) Goodreads

Goodreads - Commitment

Read More: How Udemy & Goodreads ruin my self image

Social Proof

Definition: We are wired to do things because others do them.


1) Buffalo Jump

Buffalo Jump

wikipedia: http://en.wikipedia.org/wiki/Buffalo_jump

2) Highway Traffic


3) Informercials

Social Proof - Infomercials

4) Shopify’s examples page:

Shopify - Testimonials

Read more: Social Proof: If all your friends jumped off a bridge…


Definition: We buy from people we like!


1) Avon

Avon - liking


2) Ellen & CoverGirl

Ellen - Likeable



3) Shopify

Shopify - Liking

Read more: Why that Pink Dinosaur was a Genius: Using Likeability to Persuade 


Definition: Credible people persuade us. They can be credible because of celebrity, wealth, knowledge, rank etc.


Southpark - Authority

Authority - Lil Jon

Read more: Respect my Authoritay! How Authority Helps you Convert


Definition: A persuasion technique used to build a sense of urgency (ie: limited time offer, expiry date, etc.)


scarcity - candy crush 

Scarcity - Groupon

Read More: Playing Hard to Get – Scarcity & Persuasion

So what was the point?!

The only way we’re going to learn how to be persuasive is to admit that we’re being persuaded ALL the time!

Beyond that, taking note of what persuades us and testing those same principles will give you some amazing ideas on new items to test.

Some additional reads:

  1. ConversionXL’s Post on how to use Cialdini’s 6 principles
  2. ANYTHING by the Web Psychologist

** Want to know why I created this Cheat Sheet? Taking notes doesn’t work!

So what do you think?

1)     When was the last time you were persuaded?

2)     Which principle was working in the background?

3)     How you can you use it help you convert better?

4)     What test will you start as a result?

Want a PDF version? - Download Here.

Playing Hard to Get – Scarcity & Persuasion

The last principle of Cialdini’s 6 principles of Persuasion is the idea of scarcity. If you’ve ever seen a limited time offer, expiry date, or been on Groupon for more than 10 seconds you will see scarcity in action.

Where do we see it IRL?

coupons - scarcity

Have you ever bought something you’d never buy only because you had a coupon for it?

Oh yeah.. you know what scarcity feels like.

Where do you find it on the web?

Companies like Groupon, Gilt, Beyond the Rack, and Ebay have thrived on the Scarcity Principle.


I don’t even WANT to buy a boat and the moving countdown still made me panic.

At Shopify, one of our most popular campaigns is the Build a Business Competition. We host this competition alongside Huffington Post.We ask people to build a business in 8 months. The top nine contestants with the highest sales receive $50,000; a trip to New York City to meet with mentors like Gary Vaynerchuk, Tim Ferris, and Mark Cuban and a Shopify POS package (complete with an iPad), enabling them to sell to their customers face-to-face.

The catch? You only have 8 months!

Not only does this bring many people through the door but seeing as its a limited time offer pushes our customers to their entrepreneurial limits.

Think it’s impossible?

It’s not… we have the social proof to prove it.

A lil’ Exercise

Grab a paper & pen, evernote or Google doc and jot down some ideas to the following questions:

  1. What are the most common ways you’re compelled to act FAST?
  2. What promotions, campaigns, and pages can you use it with your business?

Want to Learn More?





Respect my Authoritay! How Authority Helps you Convert

The word “authority” may instantly make you think of a boss like figure, celebrity, or Dr. Oz.

If you’re like me (and awesome), then you will probably think Cartman from South Park:

Southpark - Authority

But what does Authority actually mean?

Authority is the act of being persuaded by others who seem more legitimate then you. They seem more authoritive because of celebrity status, credentials, wealth, status or other various reasons.

Shock Therapy Never Hurt No One.. Errr…

People will tend to obey authority figures, even if they are asked to perform objectionable acts. That’s exactly what Stanley Milgram learned after conducting a study back in 1961 where he measured the willingness of participants to perform acts that conflicted with their own judgement. He was hoping to uncover the reason why so many were willing to play a part in torture during Hilter’s regime:

The result? Surprisingly people were willing to shock a complete stranger even though they knew it was horrible.

Even though people could see that someone was in “pain” on the other end of the glass, they still performed the shock 60% of the time. 

What does that say about us? 

We can be compelled VERY easily… and we have a lot more authoritative power than we think. 

How Does Authority Work on the Web?

Newsflash: You’re experiencing it! 

Just reading this blog may make me seem smarter than you (hah. jokes on you!). The fact that I may speak at a conference or hold a certain position in your eyes may make me seem more important than I actually am. 

“Well that’s silly” you say, “You’re a bum! I totally don’t believe anything you say! I’m even going to stop reading this JUST to prove it to you”

Ahem… welcome back.

Now, think of the industry people you follow. Is there a blog you read faithfully? What if you met the author in real life? Would you be nervous?

I was. 

Yesterday I had a scheduled appointment with Peep Laja from Conversion XL. I have to admit, I was super nervous. I sat nervously while he inspected my site sick at the idea he might call me out for being a fraud. 

Why was I nervous? I read his blog all the time! 

In fact, my whole talk  at #TLE2014 and journey into conversion rate optimization derived from his post on  how to use Cialdini’s 6 principles. I was so impressed by it I read everything I knew about persuasion and a year later I was speaking to the same event as him! 

Was He Horrible?

He was just as awesome as planned, and humble to boot. Just as authoritative in my eyes, but some of the scariness wore off. Now one more “celebrity” to go.. Princess Ariel/Michelle Tanner from Whatshouldwecallme,  I’m looking at you! 

So, What’s the point?

You can be an authoritative figure to those who share your job OR can make your company the leading provider of whatever… it’s your job to elicit the same emotions that you get whenever you see those “celebrities” in your world. 

Lil’ Exercise

Grab a pen & paper, evernote page or Google doc and answer 1 of all of the following questions:

  1. List all the people/companies you are afraid of / think of as an authority (Seriously, dig deep.)
  2. What makes them scary? What makes them an authority figure in your eyes?
  3. What makes them seem bigger and better than you?
  4. What can you do to R&D (Rip off & Duplicate) it?
  5. TEST IT!


Want to learn more?

Why that Pink Dinosaur was a Genius: Using Likeability to Persuade

i love you you love me with barney

 When you think about it – Barney is actually brilliant.

Seriously.. hear me out.

At the end of every show Barney made kids gather round and proclaim their love to him. He then reciprocated it back.

In his “liking” principle, Cialdini explains that people are persuaded or buy from people they like.

This isn’t old news. We were doing this back around 1950 when Earl Tupper introduced his popular Poly T products (known to you as Tupperware). Back then one of Tupper’s saleswomen Brownie Wise seemed to be selling more than anyone else. A recent divorcee and someone with no previous sales experience, learned that a home party was best way to sell products.

This idea led her to be promoted to President of Marketing in 1951. (Go Brownie!)

If for some reason you have never heard of a Tupperware party or want to watch a really sweet vintage commercial then check out this video:


Where is likeability used on the web? #Avinash


Likeability - Persuasion - Avinash

Someone who I think encompasses all that is “likeable” on the web is Avinash Kaushik. His transparency, his knowledge and most of all his humor are all elements that make him one of the most likeable and delightful men on the internet.

Don’t believe me? Watch this video:

Having a celebrity endorsement or charismatic spokesperson is one thing – but being likeable can be found in the tone of your marketing materials or your company personality as a whole.

How can you tell if a company is likeable? Take this test!

Which is more likeable the logo to the left or to the right… why?


likeability test - IBM       vs.    Likeability test - Apple

LIkeability test - Virgin Airlines  vs. LIkeability - American Airlines


Which are more likeable for you? Why?

Lil’ Exercise:

1) List some of your favorite brands in one column, and 3 adjectives that describes them on another. In the third column, ask yourself why?

2) How can your brand do the same?

3) Come up with at least 4 new tests using likeability

Comment below with some of your answers!

Social Proof: If all your friends jumped off a bridge…

“If all your friends jumped off a bridge… would you do it too?”

Social Proof - Jumping off a bridge


According to Robert Ciadini’s Social Proof theory, the answer is probably yes.

People will do things that they see other people are doing. It’s the reason we have testimonials, and review sites. It’s why you ask your friend if they’d choose Apple or PC, or Samsung or iPhone.

We need to know everyone else is doing. 


We are risk adverse.


We don’t want to look like morons in front of our friends.

Where do we see Social Proof in Real life?

Ummm.. everywhere?

We ask people for their opinions All day long..  And let’s be honest people LOVE giving it. (Even if we don’t really want to hear it.)

Giving opinions - Social Proof


Where do we see Social Proof on the web?

Social Proof - Facebook Like

Need i say more?

In case i do:

  • Social buttons
  • Testimonials
  • Reviews
  • Celebrity Endorsements
  • Word of mouth << Still VERY important don’t forget this!

Even showing people the number of customers you have or examples of them using your product are all ways we use social proof to persuade.

Lil’ Exercise:

Think of all the websites/software you use on a regular basis (Evernote, Basecamp, Facebook, Candy Crush, Twitter) and answer the following questions:

  1. What made you signup?
  2. Did you have any friends that signed up?
  3. Did you read any reviews?
  4. What things do they have on their signup page that you’re missing on yours?

Learn More

Suggested Articles:

Want to listen to something?