Netflix Ice: Validating a Need

Barb Darrow over at GigaOm is making quite a name for herself as a thought-leader in the cloud space. She has authored numerous articles over the past few months on some of the challenges faced in the cloud, including Security worries fade as hurdle to cloud adoption; Cloud wars: what happens when one vendor outs another?; GAO says “not so fast” on proposed secret Amazon-CIA cloud; and The week in cloud: FUD and loathing edition.

In one of her recent articles, Netflix dives into AWS usage monitoring with Ice, Barb writes about Netflix’s open-source tool, Ice, used internally to monitor their AWS infrastructure. She then reaches out to some of the vendors in the space, including CloudCheckr, with a request for comment. My co-founder Aaron Klein responded earlier this morning, but I wanted to reach out and respond with my own insight and thoughts.

We all know Netflix is the leading site for streaming movies and many of us still look forward to getting those red envelopes with the physical DVDs in the mail. I’ve been a Netflix customer for many years (currently only with their streaming package). But what not everyone knows is that Netflix is also the biggest poster child for Amazon Web Services. They run their entire infrastructure on AWS. As well, Netflix has been quietly building up a growing repository on GitHub of around 31 open-source projects. They use a fun set of images for their open-source projects, mirroring the movie-themed interface of their consumer-facing site, labelling the different projects with names such as “Thrillers,” “Foreign Comedies,” and “Drama.”

Notably, Netflix has put its money where its mouth is and has set aside $100,000 in prize money for innovations around its open source projects. The company is clearly not new to crowdsourcing. As a matter of fact, Netflix was one of the early pioneers in crowdsourcing. Back in 2006, the company launched a competition for its core business to improve the collaborative filter algorithm to predict user ratings for films. On September 21, 2009 Netflix awarded $1M to the winning team.

How successful will Netflix be in its crowdsourcing efforts this time? Is this open-source, crowd-sourced, or some interesting mix of the two? That remains to be seen, but the result for commercial players is surprisingly “positive.” Not positive in that it helps us directly. But positive in that it gives a thumbs up to the fact that there is a real, compelling need here. Companies clearly need tools beyond what Amazon provides to manage, optimize, and secure their AWS infrastructure.

“Companies clearly need tools beyond what Amazon provides to manage, optimize, and secure their AWS infrastructure.”

Don’t take this as a knock on Amazon. Building a compelling platform is not about providing a solution that does everything nuts to bolts. It’s rather about building an eco-system in which other vendors like CloudCheckr can add value, enabling users to choose from multiple options. Different solutions have different strengths, and those different strengths meet different users needs.

We can compare this to other similiar markets such as “IT infrastructure monitoring” for the data center. Most operating systems come with tools for performance monitoring. Linux has a plethora of methods to monitor, such as syslog. Windows includes a decent tool called PerfMon. There are multiple open-source monitoring tools out there such as Nagios (backed by a community of over a million people across the globe). Yet there is a healthy eco-system of commercial products that continues to grow around monitoring. The entry of open-source solutions is rarely a negative for the vendors in the space.

“The unquestionable truth is that people using AWS today are not using it in the most optimized ways possible.”

As a serial entrepreneur, one of the most important skills I have had to hone is the ability to uncover emerging markets. Entering an over-crowded space is tricky. Enter a space too early and you burn all your investment before the market is ready to buy. Seeing Netflix Ice enter the space is a huge validation that this is the right space at the right time for the vendors here. The unquestionable truth is that people using AWS today are not using it in the most optimized ways possible. They are still better off than if they had built their own IT infrastructure. But they are making mistakes in the cloud and they don’t have the visibility they need. Netflix Ice will help more people realize they need to get more sophisticated in managing their AWS infrastructure, which will drive more AWS users to look at the commercial products out there, including CloudCheckr. Some will stick with open-source, some will use both, but it’s guaranteed a whole bunch are going to love and need the unique value the commercial tools offer.

Follow @aaronnewman, @CloudCheckr, @NetflixOSS, Adrian CockCroft (@adrianco), and Barb Darrow (@gigabarb) on twitter.

Aaron Newman is the CEO and visionary behind CloudCheckr. He is a serial entrepreneur, having previously founded Techrigy, Application Security Inc, and DbSecure.

Keep up with the Latest in Cloud

Check out our Resources Center for cloud industry news, research, webinars, and more.