Netflix is offering a million dollars to anyone who can improve the accuracy of their movie recommendation engine by 10% or more.

Of late, a lot of companies are trying to experiment with models which depend on work being done outside the company walls by volunteers and low-paid amateurs in their spare time, mostly in expectation of a reward and recognition (from Eli Lily funded Innocentive, a intermediary which brings together a large group of experts and companies with specific research problems, for corporate R&D to Threadless.com for t-shirt design)

However, the Netflix challenge is probably the first example of crowdsourcing a tough analytics problem, rather than relying on a consulting company or in-house statisticians to build the model. While there has been no winning solution to the Netflix challenge as yet, the question remains, whether it is a viable model to consider for tough analytical problems within the company?

Karim Lakhani (link to his blog) and his team at Harvard Business School have been studying this phenomenon in the context of scientific problem solving (working paper here) based on data from Innocentive’s winning entries (30% of all problems on Innocentive have been solved). Below are some of their findings of the study followed by my commentary on its applicability to analytics:

Those problems were more likely to be solved which had a presence of heterogeneous scientific interests amongst scientists submitting solutions.
In short, diversity of problem solvers area of expertise was key. Analytics lends well to diversity as it is a very multi-disciplinary field with potential applications from a various branches of science i.e. economics, mathematics, engineering, psychology, operations research etc.

72.5% of winning solvers stated that their submissions were partially or fully based on previously developed solutions,
There is some significant research work in analytics that goes on in the universities and even some companies, and there are a lot of models and techniques out there looking or applications.

In addition the study also analyzes the winning solvers and found that:

Probability of being a winning solver is significantly and positively correlated with both a desire to win the award money and intrinsic motivations like enjoying problem solving and cracking a tough problem

Having free time to actually participate in the problem solving effort significantly and positively correlates with being a winning solver.

Both these aspects are essential for any open-source model to work.

The thing to note in the study is that companies only post those problems to Innocentive, where the internal R&D team has not been able to come up with a solution. So these are really tough problems. And that is an interesting point, because crowdsourcing a simple problem is not efficient. There is much less control over time frames and the overhead associated with managing the process will not make it worthwhile.

In conclusion:

1. An open-source strategy for solving tough analytical problems is certainly worth exploring for companies and initial research suggests that analytical problems can be good candidates for this model.
2. There is an opportunity for an intermediary (like Innocentive) to develop an open-source community for analytics.

Majority of the organizations still spend a lot of their time on data integrity issues, struggling to organize information to generate insights and not yet focused on using data to recommend actions. However, that will change over time.

This will be an interesting area to watch over the next couple of years.

Also of interest: Link to the original article introducing crowdsourcing from Wired magazine

As companies start moving from historical business performance reporting to predictive analytics, they are facing an acute shortage of skilled analytics personnel. A skilled data analyst, with the training to translate a business hypothesis into an analysis and manipulate large amounts of data in a statistically correct manner to provide a recommendation, is rare specie.

Part of the problem is that there are very few educational courses which train students in business analytics. Hence the recent announcement by SAS and North Carolina State University to develop a program for Master of Science in Analytics (MSA) degree is a welcome step in the right direction. The objectives of the program are:

• Provide students with an understanding of basic concepts and methodologies in the analysis of massive data sets.
• Show how these methods are applied to a variety of complex problems facing organizations, using real-world problems.
• Give students a sense of the broader context, such as security, privacy and ethical issues in the use of personal and confidential data.
• Provide direct hands-on experience using industry-standard software tools during a four-week “boot camp” module.

Although focus on hypothesis driven approach is not explicitly mentioned in the curriculum, in our experience every piece of analysis should be conducted with the goal of proving or disproving a specific hypothesis. By doing this the analyst will be ensuring his/her work is not in vain and it always ties back to the business problem they are trying to solve. Data analysts might not be responsible for coming up with the hypotheses, but their involvement during the hypotheses development phase is vital. This helps them understand the business problem and effectively translate the hypotheses into an analysis design and run meaningful queries.

Defining the problem and understanding the underlying data is necessary prior to selecting the statistical measure to employ in any data analytics exercise. Failing to do so will result in faulty policies, assertions and decisions as you see in the examples below:

Virginia newspaper reported that teachers were refusing to give students zeros for work not turned in because the zero had a devastating impact on the students’ average score (Via Huffington Post). They were using mean instead of median as the measure of central tendency. (Updated per Zach’s comments)

American Farm Bureau Federation added up all personal food expenses and divided by personal income to come up with the Feb. 6 date as National Food Check-Out Day by which, “the average American earns enough disposable income to pay for his or her food supply for the entire year” (Via WSJ online- subscription required). Again used mean instead of median as the measure of central tendency.(Updated per Zach’s comments)

A creationist publishes a book under the pen-name of John Woodmorappe explaining how Noah’s ark could have actually held all of the animals. He did this by computing the median size of a species, and then multiplying that by the number of species (Via Good Math, Bad Math). Used median instead of mean to define the population and extrapolate.

And finally a relevant quote:

“When she told me I was average, she was just being mean”. (credits – Gary Ramseyer)