The web as one big R&D department

Back to that web executive, the truth is that as much as he bemoans a lack of vendor innovation, his company would probably have built its own software anyhow — because that’s what big web companies do. Their real value to Hadoop vendors isn’t as customers but as R&D departments. They’re the ones doing the really interesting work around Hadoop right now, but they have limited interest in seeing any of it become commercial software of any sort.

Facebook, Twitter, LinkedIn, Netflix, Yahoo and even Airbnb are all building some significant technologies — interactive SQL engines, graph engines, stream-processing engines, schedulers, cloud-based tools. Even some startup big data vendors such as Continuuity, WibiData and Mesosphere, whose founders cut their teeth in large web shops, are releasing open source software.

Occasionally these technologies become Apache projects, but often the code is just dumped into Github or some other online repository. It’s scattered around the web, all related but often disconnected, like a rock star’s kids. If these technologies advance, it’s within the echo chamber of these same companies as engineers mingle at events throughout Silicon Valley.

I think commercializing these projects presents a huge opportunity for someone brave enough to try. The code is out there and at least some version of it is already running in production in a cutting-edge environment (that’s what made Yahoo such a valuable contributor to Apache Hadoop during its formative years). I’ve heard of big mainstream companies asking these web companies to send their engineers in and train their IT staff on these technologies. So it seems like there’s demand.

Already, the only thing most Hadoop distributions have in common are the core Apache components like MapReduce, HDFS, YARN, HBase, ZooKeeper and so on. So why wouldn’t a vendor try capitalize on the work of the Hadoop user community by grabbing the best stuff, forking it and turning it into revenue? It probably won’t be technically easy, but it has to be easier than starting from scratch and is certainly better than doing nothing.