Occam’s Razor is one of the most well-known blogs and resources in the digital analytics industry. It’s probably because its author, Avinash Kaushik, is the godfather of digital analytics. When he began his career, there were no phones, tablets or cross-channel marketing. Fast forward to 2016, and digital analytics now encompasses mobile devices and households that average eight different devices. Tracking and measuring your consumer isn’t so clear-cut anymore, and the amount of data that is out there is immeasurable. How do you approach that?
the Cliff’s Notes
In his blog post from April 4, “A Great Analysts’ Best Friends: Skepticism & Wisdom”, Kaushik talks about this surge in data and the resulting “explosion of experts” in the field. His post is a plea to be better analysts by virtue of skepticism and wisdom, as the post title would imply. Kaushik’s posts are typically lengthy, and this one is no exception, so I’ve summarized his key points for you below.
- Be skeptical of the data that is presented to you. Question the data source and whether or not these data points are really correlated. To make his point, he cites a report by The Economist studying the link between ice-cream consumption and PISA reading scores by country. Upon first glance, the data suggests causation: the more ice cream you eat, the better your reading score (how I so wish this to be true!). However, a range of other socio-economic factors that, when considered in relation to the data points, would imply that correlation is much more likely. As does common sense.
- When reviewing data like in the example above, look for sample bias. Was there an assumption made in doing the analysis?
- Statistically, did the data collected and used have all the right boundaries and controls? In other words, is it reproducible?
- Play the devil’s advocate and look for the alternative, or try to test against the hypothesis.
- Don’t get paralyzed by the data. Once you start looking behind the curtain, it’s easy to be terrified by the way data is collected. An educated mistake is better than no action at all.
- When you have made a conclusion, deliver the insights and recommendations in a timely manner so that you can impact business
- Solve for wisdom – give more than just data. Drive business decisions and change minds.
a panda’s thoughts
As I mentioned above, this was a long blog post with a lot of pieces—in talking about too much data, Kaushik may have provided a lot of words! While I agree with many of his points, I have some skepticism about others. After all, he encourages skepticism!
On his points about data paralysis, I wholeheartedly agree. As someone who designs analytics implementations for our clients, I find myself in a personal struggle sometimes. How should I set this up so that it will be useful? Sometimes we have the luxury of implementing a tool like Adobe Analytics from scratch for a client. Having a blank canvas to work on is an exciting anomaly in the analytics world. At previous companies, I’ve had to work with implementations that were messy, complicated and at times useless, resulting in data we couldn’t trust.
His emphasis on the timely delivery of analysis and insights, however, gives me pause. I challenge his quote, “An educated mistake is better than no action at all”—this depends on your audience, what exactly that data is and how it will be perceived and acted upon. In college, I had a similar motto of “done is better than good,” and I have my degree as a result…but with data and analytics, this motto doesn’t hold water.
Kaushik gives a 100/80/60/40/20 percent certainty example of the different recommendations you would give depending on the percentage of skepticism you have over the data. The problem with this rule is the repercussions of the insights or recommendations given with any level of certainty under 100%. It depends on how it will be used to influence decisions, and how any uncertainty you have may harm the trust the client has established in your expertise.
Kaushik suggests, “Avoid being paralyzed by perfection to make a decision that reflects your certainty of the data because the business needs timely decision making.” This works for some business decisions, but not for all of them. Again, know your audience and the potential impact of your insights. When talking about data, it’s hard to recover from mistakes, so be certain it’s right. With analytics, I think the preferred motto to follow is “quality over quantity.”
how we make sense of data
In our effort to help clients make great business decisions based on data, here are some of the questions my team and I ask ourselves before, during and after collecting it.
- Is the goal of this data collection to merely report, or can it help in making business decisions? How can we make it valuable?
- Will my client understand how to use this information if I’m no longer there supporting them?
- Is the data able to answer our clients’ frequently asked questions? What are their expectations for data, reporting, analysis and insights?
- What do I want to dig into more deeply? We’re hired for our expertise, and sometimes clients don’t know what they can or should look at. It’s our job to help them think about the bigger picture that data can provide.