Case study

Show me what I’m looking for

Background and task

Vodafone UK’s home page was made up almost entirely of promotional slots, promoting a single product or service.  Most visitors engaged with the navigation.  I proposed a test on Vodafone’s home page to show buyers what they were looking for.

Insight

Analytics

Site analytics data showed that at least 70% of buyers and browsers (of a pay monthly plan with a phone) started their journey by clicking the phone links in the main site navigation.  This increased to 80% + for buyers of the latest top model phones.

Desk research

Market research survey data showed that 70% of buyers(of a pay monthly plan with a phone) started with the phone before looking at plans for that phone.  Which meant  that 70% of buyers were following a phone first journey.

User research

User research showed that some buyers were getting lost when they searched for phones – especially people who didn’t know what model they wanted yet.

Most buyers want to look at phones first, when they’re buying a pay monthly plan with a phone.

“The home page needs to make it obvious how to get to wherever I want – assuming it’s somewhere on the site”

Steve Krug
‘Don’t make me think’

Proposals

home_page_phones_in_navigation_proposal
Initial proposals to put  ‘phones’ in the top navigation or a ‘see all phones’ button at the top of the Vodafone home page, got a no.
Happily, the proposal to run AB and MVT tests on Vodafone’s home page, beneath the main banner got a ‘yes’.  The  aim was to scientifically measure if a ‘see all phones’ call to action would improve engagement and sales.

Test hypothesis

A highly visible ‘See all phones’ call to action on Vodafone’s home page will increase click through rate, engagement, click to basket and click to sale*.

* by buyers of a pay monthly plan with a phone

Designs

I mocked up variants for an ABn and MVT tests.
The idea was to test the hypothesis that an obvious ‘See all phones’ call to action on Vodafone’s home page would increase engagement and sales.
home_page_proposed_see_all_phones_test_-_variant_b
home_page_proposed_see_all_phones_test_-_variant_c
A significant number of visitors knew which model they wanted, so I wanted to test showing the top selling models too.
This version added a phone picker drop down
home_page_proposed_see_all_phones_test_-_variant_d

Design compromise and live test

vodafone_homepage_live_test
The design was compromised a lot.  But at least we launched a test to scientifically measure whether or not the ‘phones’ call to action would make a difference.
This was run as an AB test on the Vodafone home page.

Results

0 %
SALES INCREASE

The ‘See all phones’ call to action element increased home page click to sale by 5%.   The hypothesis was proven with 99.9% confidence. . . that showing people what they were looking for would increase engagement and sales.   The other 2 elements (phone picker and plan call to action) made no difference and didn’t reach scientific confidence.