Atlas Recall: usability study & A/B testing to drive users to open content

atlas

At-a-glance

Based on product usage data of Atlas Recall, many fewer users than expected were opening their content, despite searching for that content. I conducted a usability study and performed data analysis to understand why our users were missing out on the key value Atlas Recall provides - getting back to their content.

Findings + Business impact

We found that users had significant difficulties finding the button to open their content. This prompted a major redesign of the button and its placement on the page. A/B testing revealed this redesign was 39% more successful in getting users to open their content.

ROLES AND DELIVERABLES

  • Wrote, conducted and analyzed usability study
  • Conducted A/B test with two different builds with ErliBird
  • Shared results with the team

TOOLS and techniques

  • In-person usability study with 6 participants
  • A/B testing
  • Data analysis with MySQL

In-person usability study

atlas-usability-study.PNG

We set out to answer the question: can users find the button to open their content?

With 6 participants recruited from our target audience, we were able to answer this question:

  • 3 participants were unaware they could open an item
  • 2 participants took between 10 and 20 painful seconds to find the button
  • 1 participants found the button quickly and easily

That was enough evidence to strongly suggest we had an issue. Our product designer, the amazing Justin Marx, kicked off the redesign.

 

Redesign of button & page

atlas-button-redesign

We identified several possible changes:

  • Button color
  • Button shape
  • Icon and copy
  • Button location

The button color was changed from light blue to bright green; the shape from a circle to a sharp rectangle; the icon was changed; and we added a text CTA "open".

 

A/B test with ErliBird beta applications

erlibird

The new button was ready to be tested. We used ErliBird, a beta testing service that allowed us to A/B test the product. 50 testers were given a beta build with the new button, and 50 testers were given the same beta build with the old button (the control).

The testers used Atlas Recall for one week. At the end of the week, I analyzed product usage with MySQL, comparing the two groups to see who had opened content and who had not. 

The new button design had 39% more people than the control group open content, at any point during the week. This was a huge success, and we implemented it immediately with the rest of our private beta testers.

 

Open beta launch

techcrunch.PNG

This study was part of a major UX overhaul, conducted a month before our official open beta launch (November 2, 2016). 

As a startup, it was vital that Atlas could easily communicate its value proposition to our users: getting back to their own content with a powerful search tool. Retention is based on the product's ability to keep that promise, which is only beneficial if they know how to re-open that content in the first place. Our original button was impeding that goal, and implementing the new button was a truly important moment for the team.