Recommended reading on the Netflix contest

The July 28th issue of the New York Times has a story on the longstanding (almost three years!) contest set up by Netflix, which offers a $1 million prize to anyone who can improve its movie recommendation system by 10%.  Last month a seven-person team of statisticians, machine learning experts and computer engineers finally passed the ten-percent threshold, which has triggered a frenzy of other submissions to beat this result in 30 days.

Contest rules state that a winner cannot be decided until all results are validated, and there is some drama being manufactured around another entry that claims to have a hair-width’s better model result, but you can read the article for that take on the story, and who will likely be getting their fifteen minutes of fame in front of a big cardboard check very shortly.  While this will no doubt create some jealous moments in our office and every other analytics shop, my take on this contest is from a couple of different perspectives.

First, I worry about the optics of an analytics contest that has taken three years and unprecedented levels of industry brainpower to produce a ten percent improvement  in the recommendations currently made by Netflix’s own internal software. Now I know from many of my esteemed colleagues here at Fulcrum that a ten-percent overall improvement in model peformance can actually be a very impressive result, but from a layman’s perspective, doesn’t that seem to be a little weak for three years of diving deep into the data?  If you’re a marketer out there looking to improve your database marketing efforts today, this is not the story you’re sending over to your CEO to help build your case for more budget.

Second, I’ve never been entirely comfortable with the criteria for this contest.  As I understand it, the ten-percent improvement is defined by predicted versus actual one-through-five-star ratings by customers.  What’s implied is that a customer that is more satisfied with the recommendations they receive will do more business with that company, which as marketers we know is not always true.  Let’s hope that Netflix, after all the PR and industry attention that it has garnered from this contest, will share some of the actual business results from deploying its new and improved recommendation engine.   A demonstrated tangible business pay-off on such a high-profile project like this would make some of those lofty claims of widespread adoption of similar automated systems for other products and services a lot more credible.  And help those of us in the industry challenge the perception among many on the buyer’s side that this kind of engagement marketing isn’t worth the time or money.   As a marketing practitioner, that’s the million-dollar prize for me….

Comments are closed.