Matthew Moore Design

At Onepager, our team often has differing opinions of the priority of planned features. To shed more light on what the correct path is, we recently ran an experiment. We wanted to see if our users would want multiple pages on their sites. So instead of taking a few weeks to rewrite the whole app to support more pages, we ran an experiment.

Creating the Illusion

We have a very solid idea of how we’ll implement multiple pages, so we designed and built out how the user will get to that feature. When the user tries to add another page, they are given a message explaining the experiment and that the feature isn’t yet available.

What to do with the data

Collecting data is one thing. Drawing useful conclusions from the data is another. I always try to be skeptical of the data being presented to me, so that made this particular experiment hard to validate.

1. Were people just clicking the button to see what happens when in reality they don’t care about multiple pages?
2. What’s a good threshold percentage to say, “Yes, we need to shift all focus to building this out now?”
3. Should we base our calculations on all users who created a page or just those that opened the Pages tab?

Once you start having so many questions and so many variables presented, your answers and conclusions start to feel like a guess themselves.

Ways to improve

In the future, we’ll have the overlay message have a brief survey about the user’s thinking when clicking the button. It could be as simple as two buttons “Yes, I’d love multiple pages” or “No, I was just clicking to see what happened”. That data would be a little easier to understand.

Either way, collecting this sort of data is a good way to inform your decisions. Just don’t go overboard with it or you’ll be creating a lot of frustrated users if every button pressed gives them an apology.