The Lean Analytics Cycle

The Lean Analytics Cycle aims to help you create a sustainable way to pick metrics that matter by tying them to fundamental business problems, creating hypotheses you can test and driving change in the business from the learning you identify.

The Lean Analytics Cycle is a simple, four-step process that shows you how to improve a part of your business.
First, you figure out what you want to improve; then you create an experiment; then you run the experiment; then you measure the results and decide what to do.
The cycle combines concepts from the world of Lean Startup — which is all about continuous, iterative improvement — with analytics fundamentals. It helps you to amplify what’s proven to work, throw away what isn’t, and tweak the goal-posts when data indicates that they may be in the wrong place.
Here's a pictorial representation of the complete lean analytics cycle:

[Note: Due to the limited screen real estate, I've redone all the images you see in this post. They preserve almost all original intent, but if you read the book, or see the cycle elsewhere, please don't be surprised to see a slightly different version. -Avinash]
While the process above might seem complex, we can simplify it to four key steps that any business of any size can apply to their analytics practice.
Step 1: Figure out what to improve.
One thing the cycle can't do is help you understand your own business. That part is your job. You need to know what the most important aspect of your business is, and how you need to change it.

• Perhaps it's an increase in your conversion rate;
• Or a higher number of visitors who sign up;
• Or a greater number of people who share content with one another;
• Or a lower monthly churn rate for users of your application;
• Maybe it's even something as simple as getting more people into your restaurant.

The point is, it's a critical metric for your business. You might need help from the business owner to figure out what the metric is. That's a good thing! It means you're relevant to the business, and if your trip around the cycle is successful, you'll have helped the organization get closer to its goals.
Another way to find the metric you want to change is to look at your business model. If you were running a lemonade stand, your business model would be a spreadsheet that showed the price of lemons and sugar; the number of people who passed by your stand; how many of them stopped to buy a drink; and what you could charge. Right there you have four things that are critical to the business and one of them is ripe for improvement. This is the one metric that matters to your business right now. You're choosing only one metric because you want to optimize it.
That metric is tied to a KPI. If it's the number of people buying, the metric is conversion rate. If it's the number of invites sent, it's virality. If it's the number of paying users who quit, it's churn.
The business model also tells you what the metric should be. You might, for example, need to sell each glass of lemonade for $5 to break even. That's your goalpost. It's the target for your KPI.
So grab a piece of paper and write down three key business metrics you'd like to change. For each of them, write down the KPI you're measuring, and what that KPI should be for you to consider your efforts a success. Do it now; we'll wait.
Step 2. Form a hypothesis.
This is where you get creative. Experiments come in all shapes and sizes:

• A marketing campaign
• The redesign of an application
• A change in the way you price things
• Building shipping costs into a purchase
• Changing how you appeal to people
• The use of a different platform
• Changing the wording on buttons
• Testing out a new feature

Whatever the case, this is where you need inspiration. You can find that inspiration in one of two ways.
If you have no data, then you can try almost anything.

• Try to understand your market. Run a survey, or look at what else they do, or examine customer feedback, or simply pick up the phone.
• Steal from your competitors. If someone is doing something well, then imitate them. It's the sincerest form of flattery. Don't be different for the sake of being different.
• Use best practices. Read up on ways that companies are growing their business, from growth hacking to content marketing, and use that as inspiration.

If you have data, then figure out what’s different about the people who are doing what you want. Let’s say, for example, that you’re trying to lower the churn rate on an application. Some of your users each month don’t quit. What do they have in common? What makes your most loyal customers different from the rest? Did they all come from the same place? Are they all buying the same things?
Either way, the hypothesis comes from getting inside the head of your audience, asking them questions, or understanding what makes them tick.
The word hypothesis means a lot of different things, but in this context I like this definition from Wikipedia the best: People refer to a trial solution to a problem as a hypothesis, often called an "educated guess”, because it provides a suggested solution based on the evidence.
We’re making an educated guess about what could improve the KPI based on what we learned in step 1.
Step 3. Create the experiment.
Once you have a hypothesis, you need to answer three questions to turn it into an experiment.

First: Who is the target audience? Everything happens because someone does something. So who are you expecting to do a thing? Is this all visitors, or just a subset of them? Are they the right audience? Can you reach them? Until you know whose behavior you’re trying to change, you can’t appeal to them.
Second: What do you want them to do? Is it clear to them exactly what it is you’re asking them to do? Are they able to do it easily, or is something getting in their way? How many of them are doing it today?
Third: Why should they do it? They’ll do what you ask if it’s worthwhile to them, and if they trust you. Are you motivating them properly? Which of your current pleas is working best? Why do they do this thing for your competitors?

On the surface, these three questions—who, what, and why—don’t seem hard to answer. But they are. That’s because they require you to have a deep understanding of your customers. In the Lean Startup world, this is called customer development. The experiment will almost always look like this:
Find out if WHO will do WHAT because WHY enough to improve KPI by the Target you've defined.
This gets to the deliberate nature of the actions we want to take. You start with a great hypothesis, and you’ll get a great experiment. This also keeps you honest, because everyone recognizes the point of the activity beforehand.
In our case studies, below, you'll see that the KPIs were things like Property Bookings, Number of Engaged Users, and Daily App Use. If you have access to existing data, take some time to document what the current performance looks like. Of course, it’s possible that you don’t have access to the data (as in the Airbnb case study below.) That’s OK. We have a path for that as well.
Once you have your experiment, set up your analytics to measure the KPI against its current baseline and the goal you’ve set. Then run your experiment.
Step 4. Measure and decide what to do.
At this point, you’ll know whether your experiment was a success. This leaves us with several options:

1. If the experiment was a success, you’re a hero. Celebrate a bit, then find the next metric that matters the most and move on to the next who, what, why exercise.
2. If the experiment failed spectacularly, we need to revisit our hypothesis. It’s time to identify a new who, what, and why, based on what we’ve learned. Remember, as long as you learn from it, failure is never a “wasted” opportunity.
3. If the experiment moved the needle, but not enough to clear the goalposts, we should try another experiment. Our hypothesis might still be valid, and we can try again, adjusting based on what we’ve learned.

The underlying beauty here is that we’re being smart, fast, and iterative. We’re making a deliberate plan, measuring its results, and circling ever closer to our goal. Identify, hypothesize, test, react. Repeat.
Let’s look at some case studies that will really help to drive the Lean Analytics Cycle home. We don’t know everything about the metrics these companies dealt with — they’re private companies, after all — and in some cases we’ve estimated numbers to make the explanations more clear. But even with these changes, the examples will help make all of this a bit more real.
Case Study 1: Airbnb
Airbnb is a hugely popular marketplace for rental-by-owner properties. They’ve found dozens of creative ways to grow, but they’re always judicious and data-driven.
Step 1: Figure out what metric to improve
The metric they wanted to improve was the number of nights that a property was rented. Notice that this is more central to their business than simply measuring revenue: Airbnb does well if its homeowners do well, and for it to succeed, it needs listed properties to be rented often so the homeowners will stick around.
The company knew that to succeed, they needed a significant improvement in rental rates per property.

• One Metric That Matters: “Number of nights rented.”
• KPI: Property bookings
• Target: (not publicly known)
• Current level: (not publicly known)

Step 2: Form a hypothesis
We don’t know how Airbnb came up with its hypothesis. But we know it had access to property listings that rented well.

• Perhaps they had noticed that the pictures of those properties looked, to them, more professional.
• • Maybe they realized that one common complaint from renters was that the property didn’t actually look like the pictures on the site.
• Maybe they found that people would most often abandon a listing right after seeing photographs.
• Maybe they analyzed the metadata from pictures and found that there was a strong correlation between properties that rented often and expensive camera models.

However they got there, they formed a hypothesis: Properties with better pictures rent more often.
Step 3: Create the experiment
Armed with the hypothesis, it was time to create the experiment. As is often the case, having a clear hypothesis makes devising the experiment fairly easy. Their who, what, and why are as follows:

Who is the campaign for? Travelers looking at listings on Airbnb. What do you want them to do? Decide to rent a property more frequently.
Why do they do it? Because the photographs look professional and make the property look beautiful.

So for them, the experiment was
Find out if travelers will book more properties because of professionally photographed listings enough to improve the property bookings by X%.
Notice that in this case, Airbnb didn’t really need any current data. It might just as easily have been a random comment over lunch that led to the hypothesis. But even if the hypothesis isn’t founded in hard data, the experiment design must be.
To run this experiment, Airbnb created what Lean Startup calls a curated minimum viable product. This is like the Wizard of Oz: most of the hard work is done behind the curtains, but the end user thinks they’re seeing a final product.
Airbnb wasn’t sure whether or not the experiment would work, so the team didn’t want to hire a staff of photographers or invest in a new part of the application. But at the same time, they had to have a real test of an actual feature.
Before we go further, there’s an important lesson to take away here. You can do this at Nordstroms, or Expedia, or Unilever. You don’t need to build a magnificent shining castle. You don’t need a beautiful beast to go out and test. You can start small, lean, and mean — with just the customer-facing pieces you want to test — and go validate (or disprove!) your hypothesis.
Airbnb’s experiment consisted of something that looked like a real feature, but under the covers was really just humans and contracted photographers. During the experiment they took pictures of properties, and then measured the KPI, comparing properties that had been photographed to those that hadn’t.

Step 4. Measure performance.
In this case Airbnb measured the bookings from the few properties that had professional photos and compared the rate of bookings with properties that only had photos taken by property owners. The result? The properties with professional photography had 2-3 times the number of bookings!
[Remember that the raw number is not the only important part, we would also measure statistical significance. Airbnb had enough data points to be confident in their results. -Avinash]
By 2011, the company had 20 full-time photographers on staff.

The graph is impressive, right? There were many other things going right with Airbnb's business and business model. But the lean process was a key contributor to improving the bookings rate. Clearly, the experiment was a success. They celebrated a bit, then went on to fix the next biggest problem in the business.
Case Study 2: Circle of Friends
Circle of Friends was a social community built atop Facebook that launched in 2007. It essentially allowed you to create a group of friends who could interact and share content, much as people do today with Google+, before such features were part of Facebook.
As a social network based on user-generated content, Circle of Friends only grew when its users were engaged. The company’s creators wanted users not only to create groups, but also to send messages within those groups, invite others, and interact with Facebook elements such as news feeds.
Step 1: Figure out what metric to improve
The founders had several measures of “engagement”, from whether people attached a picture to a post, to whether they clicked on Facebook notifications, to the length of posts they wrote. Any of these actions — and several others — constituted “engagement.” All of this rolled up into a simple KPI: number of engaged users.
While Circle of Friends’ launch was hugely successful in terms of raw attention — they had 10 million users! — engagement simply wasn’t happening. None of the metrics they used to measure the health of their communities was where they wanted it to be.
The rather nebulous “level of engagement” was a compound metric, which can be dangerous to rely on. But one of the clear metrics they tracked was “number of circles with activity”, meaning that there had been some form of interaction within a group in the past few days.

• One Metric That Matters: User engagement
• KPI: Number of active users; number of circles with activity
• Target: (not publicly known)
• Current level: Less than 20% of circles had any activity after creation

The challenge, here, was how to kickstart engagement. That’s not an easy thing to do. Fortunately, Circle of Friends had plenty of user data to mine.
Step 2: Form a hypothesis
Circle of Friends had huge volumes of information on its users and how they used the product. They looked at two groups of users: those that were engaged, and those that weren’t. Then they looked at what those users had in common. In other words, they defined what engagement meant (i.e. “someone who has returned in the last week”) and segmented users into two groups. Then they looked at other things those people had in common.
What they found changed their business. It turned out that the engaged users were much more likely to be mothers. Looking at moms who use the application:

• Messages to one another were, on average, 50% longer
• 115% more likely to attach a picture to a post they wrote
• 110% more likely to engage in a threaded (i.e. deep) conversation
• Those who were friends of the circle owner were 50% more likely to engage
• 75% more likely to click on Facebook notifications
• 180% more likely to click on Facebook news feed items

Before we look at what they decided to do, stop for a minute and realize that simply having data isn’t that useful. Circle of Friends had all of this information. But asking the right questions of your data is a superpower. In this case, they asked, “what do engaged users have in common?” and it absolutely changed the destiny of their company.
From their data, they formed a hypothesis: If we focus only on moms we’ll have the engagement we need.
Step 3: Create the experiment
Their who, what, and why were as follows:

Who is the campaign for? Moms. What do you want them to do? Engage with other users, invite new users, and create content.
Why do they do it? Because they find interaction with others rewarding and compelling.

In the end, the founders made a huge bet — the kind of bet that only startups with nothing to lose can make. They decided to completely rebrand the company as Circle of Moms, and focus everything on attracting and engaging mothers.
So their hypothesis was:
Find out if moms will join, relate and engage in groups because of a community targeted specifically at them enough to improve the number of active circles and engaged users by X%.
They were betting that if they stopped trying to please everyone, and instead focused specifically on mothers, they’d have a higher percentage of engagement, which would in turn lead to the company’s overall success.

Step 4. Measure performance
It took a few months to carry out the experiment, since it involved re-launching the entire company. The initial result was a huge drop in users, of course, since most of the communities weren't made up of mothers.
They lost millions of users.
They feared this would happen, but they knew that if they didn't get engagement where it needed to be they might as well close up shop.
If they'd been focusing on the wrong KPI, such as number of subscribers, they'd have quickly concluded that the experiment was a disaster. But because they'd been deliberate about their hypothesis, they knew that the most important metric was engagement. Sure enough, the users that remained were engaged. Overall engagement and active circles climbed significantly, to the point where the business model was healthy.
Of course, having achieved the engagement they sought, the next experiment was about whether they could grow a more narrowly focused community to a decent size. By late 2009, that experiment was a success, too; they'd climbed back up to 4.5 million users, with strong engagement.
Case Study 3: High Score House
High Score House is a tool that lets parents manage household chores and rewards for their kids. The founders envisioned a world where parents and children sat down each day to identify chores, keep score of what the kids have done, and drool over rewards they might claim.
Step 1: Figure out what metric to improve
In the early stages of launch, the most important metric was the number of families who fired up the app at least once a day. Those who did were considered "active", while those who didn't were assumed to have abandoned the application. Unfortunately, while High Score House was getting installations and great press reviews, users weren't active.
No active users, no business.

• One Metric That Matters: "Active families"
• KPI: Percent of families who've used the app in the last 24 hours (AKA "percent daily use")
• Target: Over 60%
• Current level: Below 20%

Step 2: Form a hypothesis
The founders went around the Lean Analytics Cycle several times:

• They tried changing the look and feel of the application.
• They tried sending people messages urging them to come back to the application.
• They tried adding new features.

None of it worked; fewer than 20 percent of families who were part of the early release used the app each day.
Each time, their hypothesis went something like, if we send a reminder email to parents each day, the percent daily use will exceed 60%.
Step 3: Create the experiment
The High Score House team went dutifully through the motions of the Lean Analytics Cycle. For the notification hypothesis, the who, what, why looked like this:

Who is the campaign for? Current installed base of users. What do you want them to do? Use the app daily.
Why do they do it? An email will remind them to launch the app and sit with their kids, making it a daily activity that becomes part of their routine.

In other words, their experiment (for the notification model) was:
Find out if parents will use the app daily because we remind them to do so enough get the percent daily use to 60%.
They ran the experiment.

Step 4. Measure performance
The experiment disproved the hypothesis. Email reminders didn’t measurably improve percent daily use. Like many of the other changes the team had tried, they couldn’t move the needle.
At this point, Kyle, the founder of the company, decided it was time to ditch the data and go qualitative. He picked up the phone and started talking to users. And this simple, messy, unquantified, anecdotal process yielded an amazing insight:
Many families would set aside a specific day of the week to plan their chores and count their rewards.
Kyle was floored. Families were getting value from the product. They were just using it in a different way from what the designers intended. Kyle now had new information which he could use. But it wasn’t for another experiment. This time, Kyle was going to move the goalposts/target.
Changing your target should never happen lightly. It’s only OK to change the goal when the customers have given you new, validated learning from which to make the change. In this case, since your goal is tied back to your business model (remember the $5 glass of lemonade?) changing the goal means changing your business model
In this case, HSH was going to have to change the way their application was designed, making it more suitable for week-by-week views and a different usage pattern. But by moving the goalposts, the team had a new KPI: Percent of families who've used the app in the last 7 days (AKA "percent weekly use"). And with this new definition, over 80% of the early adopters were using the application regularly.
We’ve seen three case studies. The first one, about Airbnb, shows a straightforward example of a hypothesis and an experiment without data. The second, about Circle of Friends, shows an example of an experiment based on data. And the third one, about High Score House, shows how to adjust the goalposts when they aren’t set properly in the first place. All three are examples of the Lean Analytics Cycle at work.

______________________________________________________

Amazing, right? Life's pretty cool when you have a process!
For me the lean analytics cycle is, at it's very core, about driving change quickly, regardless of how much you know about your users or how much data you have. It discourages waiting for perfect data or perfect understanding of every variable or perfect understanding of what the business is trying to solve for (because in all three cases you will never get perfection!).
While it was developed with startups in mind, you can see the attraction of speed, learning and constant improvements for businesses of any size.
It is important to note that the cycle, and the Lean model as a whole, isn’t advocating chaos. They might deal with uncertainty, but they're not random.
There is great deliberation in step one to identify the KPI that will be a guiding light for us (embrace "One Metric That Matters"). There is a lot of deliberation in step two on ensuring that we have an optimal hypothesis to work from. Then figuring out how we are going to experiment with deep clarity from defining the who, what, why. Finally, understanding whether we actually succeeded by measuring performance. Then, then … you learn. You internalize. You win.
And how can you be immensely incredible at this? Go get Alistair and Ben's book Lean Analytics .

By Avinash Kaushik

Source : http://www.kaushik.net/

Posted in: Lean Analytics Cycle,Web Analytics