Big-Data Analysis Update – Hadoop Grows Up

A few months ago, I wrote a post about open-source, non-relational data platforms that were beginning to become important for marketing analysis. In particular, I singled out Hadoop as an application that we had begun to work with, as many of the data sets we deal are large. I made the following prognostication:

My view is that such technologies will move from being rather esoteric to becoming more mainstream within the next few years. With commercial support from major suppliers like IBM and Oracle, the continued development of open-source alternatives, and continued research and development by heavyweights like Google and Amazon, non-relational database technology is something that will find many marketing applications.

This spring has seen a whole slew of Hadoop-related developments, all pointing to a maturation of the application and the vendor support behind it. Here are a few highlights:

• Hadoop was originally developed at Yahoo! and has been an Apache Foundation project for several years. June saw the formation of a venture-backed for-profit company, HortonWorks, that will develop premium services and solutions around the core technology. It begins life with Yahoo! as its client (Yahoo! also has the largest current implementation of Hadoop: 200 petabytes and counting).
• There are a number of other young companies that have been extending and packaging the technology, including MapR (which recently signed a deal with EMC), Cloudera, and DataStax (which recently  released Brisk, a hybrid Hadoop-Cassandra solution). Such vendors are important to the wider adoption of the technology, as usability will improve when for-profit solutions respond to market forces.
• Support from more established vendors continues to strengthen. For example, Amazon (Fulcrum’s cloud computing platform) offers an array of Hadoop-related services.

The folks at eWeek have put together a nice slideshow that summarizes the many developments over the past few weeks. And the fact that a mainstream IT publication like eWeek is devoting this type of coverage lends extra evidence to Hadoop’s current momentum.

What does all this mean for marketers interested in trying the technology? It means that there are now a range of options, from the original “roll-your-own” approach (download the app from Apache) to applications and platform vendors, such as MapR and Amazon, to marketing analytics companies, such as Fulcrum.

For those that are ready to move from trying to buying, the growing vendor base offers real options for finding and comparing options. Moreover, the vendor and user community is building a broader set of real-world case studies, some of which will be on view at Hadoop World 2011, sponsored by Cloudera.

One Response to “Big-Data Analysis Update – Hadoop Grows Up”

1. What is a good introduction to working with large data sets? - Quora says:

   July 29, 2011 at 9:36 am

   [...] bookIf you want to learn about new noSQL type databases checkhttp://www.google.co.uk/search?q…or http://www.fulcrum-mktg.com/data…This answer .Please specify the necessary improvements. Edit Link Text Show answer summary [...]

   Reply