My Experiment with Conversion Rate Optimization

conversion rate optimization - designI recently decided to do a marketing experiment with pay-per-click (PPC) advertising. I manage a factoring company that wants to increase the size of its portfolio, and I thought that PPC would work well for us. In case you don’t know what factoring is – it helps small companies by financing their slow-paying invoices.

As you can imagine, competition in the financial services market segment is pretty fierce. A number of companies have budgets that are much higher than mine. One of them is publicly traded on the NYSE. I suspect our total revenues are lower than their marketing budget (they are a good company, BTW). Since I know that I cannot outspend them, I wanted to test if I could outsmart them. My tool of choice for this experiment was Conversion Rate Optimization, also known as CRO.

A few words about CRO

As its name states, conversion rate optimization helps you optimize your site so that you can turn more visitors into leads. The objective is to take the traffic that you already have and make the most of it. The concept can be simple to understand but hard to implement because of the many moving parts.

Getting a true feel for the real value of CRO can be hard since the market is full of companies that make outlandish claims. How many times have you seen an ad that says something like “I changed the color of my fonts (or the size, or some other little thing) in a page/ad and my conversion improved by 300!” Obviously, these claims are very hard to believe.

I decided to run a little experiment to separate fact from fiction. By the way, I am not a CRO expert – nor do I pretend to be one. I am just a small business owner who decided to test this for himself.

[Resource: You can learn more about CRO at Conversion Rate Experts]

Simple A/B testing

My method for CRO was fairly simple and used A/B testing. Basically, you split your users between two versions of a landing page and see which performs better. The pages vary by some details. It can be anything such as the color of the font, a different call to action, or the marketing message. Once you have collected enough data, you evaluate the pages and declare a winner.

From that point forward you can use the winner as your main landing page for that market segment. Many marketers use CRO as an ongoing process to create better websites.

Dealing with limited data

One of the challenges that I ran into is that clicks for my industry are very expensive. You can easily burn through a lot of money before you get statistically valid results. This can be a serious problem for small companies and leaves you with two options. The first option is to run a statistically valid test which should provide solid answers but at a very high cost. The second alternative is to run a test with limited data and use your best judgement. The results may not be perfect but are often “good enough.”

This second approach worked best for me. Most of the tests were good enough to tell me which page was better. However, due to the limited data, they could not tell me how much better very accurately.

What did I find?

I found that CRO works quite well if you are disciplined enough and if you work hard at it. I was able to generate decent results which increased conversions by 10% to 20%. It took some practice but it was doable. Most of my experiments looked like this:

regular conversion rate optimization results

However, in one case I was able to generate a conversion increase of over 100%. I was testing two pages: the old one vs. the new one. The new one proved dramatically better. What makes this test very interesting is that I was convinced that the new page would do badly – I thought it was ugly. This last point also proves that you must test all your assumptions. Let the facts speak for themselves.

Let me show you the results. Obviously, I want to keep my actual data private but I can share a modified version of the table (data has been changed):

conversion rate optimization experiment

What did I learn?

I would say that this experiment was very valuable for me and my company. As part of the process, I had to look at our website and marketing materials from a new perspective: the user. Some experiments proved that many assumptions I had about our users were plain wrong. This process was very valuable.

However, if I had to sum up everything I learned into four points, they’d be:

  1. Manage your expectations – most results will be average (at best).
  2. Results can be very unpredictable. Test to find out.
  3. Be comfortable making decisions with limited data.
  4. Learn to love math 🙂