Virtualization.com

News and insights from the vibrant world of virtualization and cloud computing

video

Amazon Loves CloudStatus, And We’ve Got The Proof on Video

by 2 Comments

At the O’Reilly Velocity conference we attended last month, Hyperic was there to hype the launch of Cloudstatus, which aspires to become the monitoring tool for cloud providers similar to what Netcraft is for’classic’ hosting providers. Get the lowdown on Hyperic and CloudStatus in this video (two parts) from Jon Travis (Principal Engineer) and Xavier Soltero (co-founder and CEO).

See part 1 of the movie here on Vimeo, but scroll down for the best part!

As we are living in the ice age of cloud computing, glitches (like the recent outage of Amazon S3) are to be expected and it must be said that Amazon managed to fix its ecosystem relatively fast and openly reported on the underlying problem .

An interesting question is posted by Reuven Cohen on his blog about the use of federated network protocols within cloud services and the gossip protocol that caused Amazon’s WS downtime on June 24.

“…We have been big fans of use of XMPP for federated communications within our Enomalism cloud platform for multi cloud communications (Wide Area Cloud). XMPP is interesting because it natively solves a number of federation problems within a tried and tested framework. One of the biggest benefits to the use of a gossip protocol lies in the the robust spread of information and the exponential nature of it’s sharing of information within a large number of machines…”

At Virtualization.com, we intend to report on cloud initiatives too, since all these Platform-as-a-Service providers (Google App Engine being the exception to this rule) are enabled by virtualization technology. We expect to see several more competitive statistical analysis tools for various cloud service providers to emerge in the near future. With Amazon Web Services (AWS) blazing the cloud trail, Hyperic picked them to start reporting on via CloudStatus. But Google App Engine and (Sales)force.com seem target platforms to follow. So Amazon’s trouble with being first, is they are first in line to be publicely reported on too. This also means the PR and sales people at Cloudstatus have a busy time issuing press releases and contacting impacted prospects whenever Amazon experiences a glitch or failure.

Stacey Higginbotham at GigaOm ventilated the common fear that:

“… Amazon or another cloud provider could shut the service down, either by offering their own status service or by stopping the Hyperic agent. Given the rush to provide dashboards, application-testing products and other services on top of established computing services, I’m eager to see how startups keep their footing in the clouds.”

Being curious and knowing Amazon only speaks through CEO Jeff Bezos or CTO Werner Vogels, we walked up to the latter and were happy to learn Amazon actually loves CloudStatus. He took a step back right afterwards, but why not just watch the video to see his response to the CloudStatus launch?


Amazon AWS Loves CloudStatus.com, Here’s The Proof (Hyperic Video 2/2) from Toon Vanagt on Vimeo.

On a sidenote: Hyperic’s newly launched CloudStatus detected the outage at 8:45am PDT, a full 20 minutes before Amazon posted that they were aware of the issue, at 9:05 PDT on http://status.aws.amazon.com/. CloudStatus saw several server errors coming from the majority of their S3 and SQS monitoring agents, in addition to other problems with EC2 (lots of EC2 zombies being created) that may have been related.

Like hurricane warning systems, while Hyperic CloudStatus could not have prevented the S3 outage, it was able to provide enough of a “storm” warning for users to take action. The company will be adding additional cloud services to CloudStatus in the coming months, next up is Google App Engine.

Hyperic

Rich Wolski on Eucalyptus: Open Source Cloud Computing (Video Interview – 2/2)

by Leave a Comment

In this second part of our video interview with Rich Wolski (see the first part here), recorded at the O’Reilly Velocity conference, we learn how Eucalyptus came around the Amazon subscription method, where credit cards are the key to authentication. Offering ‘free and open’ clouds in university environments was achieved by introducing a system administrator in between the user account request and the issuing of certificates. Upon user request, the Eucalyptus user subscription interface generates an e-mail to an administrator, who will then perform a ‘manual’ verification. This can be a phone call or a physical meeting.


Eucalyptus Director Rich Wolski on open source cloud computing, Xen and Amazon’s EC2 (part 2/2) from Toon Vanagt on Vimeo.

Users did not like Rocks (leading open source cloud management tool), but the community (in smaller community/ deployment supports) preferred to do this manually. So Eucalyptus 1.1 provides Guidance, a script to build from scratch by hand.

A ‘build with one button’ remains the goal for future versions.

The full Eucalyptus image is only 55 Mb (without Linux image) and includes the necessary packages in order to make sure all of the revision-levels are fully compatible. Eucalyptus comes as Free BSD Open-Source license with a small disclaimer that the University of Santa Barbara explicitly wants to avoid any intellectual property infringements and will take necessary steps if needed.

Virtualization is supported by Xen 3.1 for security sake (3.0 works too, but is discouraged).

Lessons learned in building clouds from open source are quite rare. Here are a few from Rich:

Unlike commercial environments (where one controls the configuration, hardware purchase and networking), the architectural decisions are very different in open source environment, where one does not know the installation. One of the current challenges is to build a system depending on the control you have over your specific installation, you could successfully remove more of the portability from the system as you needs fit.

A second lesson is that people do things by hand and this is an opportunity for automation. Nobody is deploying Linux manually, instead sys admin use distributions. Shouldn’t there be a similar cloud distribution product out there? The people at Puppet were eager to help on providing such scripts for cloud deployments. According to Rich, this illustrates how O’Reilly should be credited for creating a good atmosphere at the Velocity 08 conference where a lot of cross-fertilization happened.

Rich ends the interview by throwing a fundamental question at the cloud community. He classifies current cloud initiatives on a scale based on the ‘closeness’ of the application layer to the cloud API. At the one end of this spectrum, he puts Google Apps (with Python oriented function calls) and at the other end Amazon EC2 (a set of very simple web service interfaces to the underlying virtualization technology) and all other cloud offerings float in between. This impacts what you can do with virtualization. Google AppEngine becomes your compiler on their end of the scale.

Rich wonders if this tighter link to the Google AppEngine will become a liability or an asset in the future when it comes to virtualization capabilities?

We invite you to provide your answers in the comments below!

Exclusive Video: VMware In A Nutshell, By Product Marketing Manager John Gilmartin

by Leave a Comment

So what’s VMware all about now on a technology level, now that their stock went tumbling and their CEO has been replaced? John Gilmartin, Group Manager, Product Marketing at VMware answered our questions on virtualization in general and their strategy and product portfolio in particular.

After defining virtualization and virtual machines, John dives deeper into what drives prospective virtualization customers (infrastructure consolidation, high availability, business continuity, disaster recovery?). He also shares more on their recently announced VMsafe initiative, which allows trusted partners to ‘peak’ into a Virtual Machine and identify threats from the outside of the operating systems and the applications they try to protect.

John underlines that VMware is offering features far beyond the basic data center consolidation needs; such as life cycle management, Virtual Desktop Infrastructure (VDI), automated provisioning, fail over, disaster recovery and optimize the management tasks for a virtual infrastructure.

All this is part of VMware’s from-desktop-to-datacenter portfolio and makes VMware feel confident and well positioned to compete with Hyper-V from Microsoft.

Rich Wolski on Eucalyptus: Open Source Cloud Computing (Video Interview – 1/2)

by 4 Comments

A month ago we reported on how you can build your own open source cloud on clusters to make your personal cloud dreams come true!  Simply put your datacenter to use by ordering Xen virtualization on the Rocks and then carefully roll it in fresh Eucalyptus leaves.

In order to learn what makes these clouds tick, we have sent our enthusiastic cloud computing koala Toon Vanagt to San Francisco to interview Eucalyptus Director Rich Wolski at the O’Reilly Velocity conference. Below, you can find the first part of this exclusive video interview (we’ll post the second part tomorrow).

Rich’s students came up with EUCALYPTUS, which stands for ‘Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems’ as an open-source tool for doing “cloud computing”. Their tool is designed to stimulate the development, interest, experiments and research into the nascent concept and industry of cloud computing.

Eucalyptus was build in a modular fashion, so it can “mimic” the interface of popular commercial clouds, like the one they started off with, Amazon EC2. The team plans support for several cloud interfaces as long as they are public and well documented.

Rich underlined that Eucalyptus is designed to experiment and not to compete with industrial strength clouds as Amazon EC2. Although with some engineering, one could take parts of Eucalyptus, mature those and scale to specific needs.

When asked about the underlying virtualization experience, Xen is seen as a very useful technology in ‘cloud’ regards. Rich complements Xen on being well documented and conceptually easy to understand and he looks back at the Xen selection as a good first hypervisor choice. Due to the nature of their specific use, parts of Xen would ‘break’ under load and were modified to meet certain stability needs.

As LibVert is used, Eucalyptus should in theory be relatively easy be able to support other hypervisors such as VMware and KVM. If no major wrinkles appear on the development surface, Eucalyptus therefore expects to support VMware and KVM with its 2.0 release, scheduled for early September 2008.

Rich supportively points to the Enomalism elastic computing platform, an open source cloud platform that enables a scalable enterprise IT and local cloud infrastructure. as an alternative open source virtualization system.

Security remains an issue but in some respects, accountability and authentication are an even bigger problem to the open source community than within commercial projects. “In an academic space, where you are not paying for usage, it is not a credit card that you are accounting to, but a user”. So Eucalyptus had to devise a user accounting system that is based on certificates. On top of that components should not be ‘spoofable’, as there is no message encryption in Eucalyptus (yet). Because these messages can be spoofed, Rich’s team had to take care of an open source implementation of Web Services Security to make sure the cloud controllers cannot be ‘fooled’ by malicious messages of doubtful origin.

The shortage of public IP addresses in university environments was solved by using the open source technology VDE (Virtual Distributed Ethernet). [VDE is an ethernet compliant virtual network that can be spawned over a set of physical computer over the Internet. You can see VDE as the software incarnation of a hardware network switch plus attached cables. Using the vde_switch and vde_plug programs you are able to create quite complex virtual analogies of a network that can span several hosts, even across the Internet.

By creating a virtual Ethernet for every cluster allocation and make that a set of user space processes can tunnel through NATs (Network Address Translation). As a downside to this VDE implementation comes a performance penalty. So Eucaluptus is offered with 2 flavors, linked to the SLA-nature in cloud computing. The first option uses the described very flexible ‘Virtual VLAN’ independent of IP-addresses. The second option bypasses VDE and is faster but less scalable as limits user requests to the confines of a single cluster.

Tomorrow, we’ll publish the second part of this exclusive interview. Stay tuned!

“Benchmarking” The Citrix / XenServer Combo with Ian Pratt (Video Interview – Part 4)

by Leave a Comment

During the Fosdem 2008 conference, we had a chance to sit down (on a bench) with Xen Guru Ian Pratt. Below is the fourth and last part (see part 1, part 2 and part 3) of our exclusive interview, where Ian shines his light on Citrix Xenserver, relocating virtual machines (VM), VM-mirroring, OVF, page tables algorithms, open source community involvement, management frameworks, the Citrix take-over, Virtualization marketing with OS-enlightment, FUD-tactics by VMWare, self-healing servers, Xen embedded in firmware, why Amazon goes with Xen, the Xen GPL license, OracleVM, xVM (Sun), Parallels and the future of virtualization…

We cut the interview into 4 digestable pieces, which we publish one at a time (see part 1, part 2 and part 3). As said, this is the final part (soon, you’ll also find a written transcript below for your convenience):

The video is also up on YouTube and Steamocracy.

“Benchmarking” The Citrix / XenServer Combo with Ian Pratt (Video Interview – Part 3)

by Leave a Comment

During the Fosdem 2008 conference, we had a chance to sit down (on a bench) with Xen Guru Ian Pratt. Below is the third part (see part 1 and part 2) of our exclusive interview, where Ian shines his XenServer light on the Xen page tables algorithms, open source community involvement, management frameworks, the Citrix take-over, virtualization marketing with OS-enlightment, FUD-tactics by VMWare, …

We cut the interview into 4 digestable pieces, which we publish one at a time (see part 1 and part 2). As said, this is the third part (you can also find a written transcript below for your convenience):

This video is also available on Vimeo and Streamocracy.

(0:02) As you are one of the core members of the Xen project, you know that one of the hardest issues to address are the shadow page tables, which are a head ache when you build a hypervisor. I believe you are in the 6th rewrite of the Xen page tables algorithms. At the same time we see that the hardware vendors try to address this in a different way, by supporting it from the hardware up. What is the best way to go?

“It is one of those areas where having some hardware support certainly helps, but it is not a panacea, certainly with the hardware implementations that exist today. There are plenty of benchmarks (probably most benchmarks) that prove that the software approach of Xen wins out. Because there has been a lot of investment into that software approach and there is some really clever code in there right now, written by some super smart people. It is an interesting arms race between the two. One of the things that we are looking at is depending on the workload -dynamically chosen- whether you use the hardware approach or the purely software approach. You kind of hope that for that particular one –at least for the basic functionality- the hardware wins out over time. But there will always be parts of virtualizing the MMU (Memory Management Unit) which are best done by software. That is where OS-enlightment (aka) Para-virtualization comes in. That is a huge win for virtualizing the MMU.”

(1:30) That is a term I hear more often now. Where does the marketing term: “OS-enlightment” come from?

“We had been using the term para-virtualization. I think it was Microsoft that came up with the term “enlightment”, which we have been told is very much a nod to the Xen-heritage. Microsoft probably has rather more budget to spend on marketing than open source projects.”

We all know Microsoft understands a few things about marketing.

“I am not at all upset with that term. I am quite happy to use it and adopt it.”

(2:09) So Ian, it is quite interesting you just mentioned Citrix and the sun joined us. Do you think the contributions from the open source community have slowed down since the Citrix takeover?

“We certainly have not seen that! If you think about the life of the Xen project, there have been a number of significant changes.
When we left the university to setup XenSource, people were worried we might go off and take Xen in closed source or something, which we did not do. It is still the same group of guys, basically myself, Keir Fraser, Steve Hand, Christian Limpach…all off the same guys working on the project, with now many more off course.

Then Citrix acquired XenSource and we obviously had to explain to people what was happening. I think our community has seen that nothing has changed. One of the things that we did do was just to provide greater transparency. We have setup Xen.org, the Xen advisory board and all of the web site and everything where we run Xen.org. The advisory board now has focus from companies like Intel , AMD, HP, IBM. All big companies that are now contributing to Xen and have that oversight from the advisory board. So I think the community is pretty happy and it’s going from strength to strength.”

(3:33) How do you see the shift XenSource (now XenServer) made from building a para-virtualized platform, that served the open-source community and mainly targeted unix/linux-environments, to a company which has another main audience with Bill, the average Windows admin.

“We were never focused just on running only open source operating systems. That was never the aim. We wanted to build a platform that would be OS-agnostic and to be able to run any OS and do a great job at it. We have always put an awful lot of effort into supporting Windows, because there are a lot of windows OS instances out there, we can’t deny that. It is something that always has been important to us. What is different is the way that XenSource and now Citrix look at packaging Xen. Lots of different companies are bringing Xen to market. Obviously the Linux vendors are mainly concerned about running Linux. Solaris and Sun are mainly concerned about running Solaris. One of the things Citrix / Xenserver are trying to do is making sure it is OS agnostic and we did a great job at running Windows and a great job at running Linux as well.
Xen is awesome running Linux and completely blows any other virtualization solution out of the water and at running Windows it is extremely good too. Let’s put it this way: I am unaware of any benchmarks we lose. “

(05:02) When you look at the fight going on between the companies building the management frameworks for Xen and projects like Enomaly, OpenQrm, Redhat & Novell. Was the acquisition of Xensource by Citrix your easy way out of that fight?

“I think we are still very much in the fight. Xensource and all of these other companies are building management frameworks on top of Xen. I think that all of these companies are coming at it from a different point of view. Linux vendors are trying to provide that same look & feel they have within Linux and expose Virtualization through those same GUIs and tools. The difference is that companies like XenSource and Citrix are interested in making it very easy to use and are building a Virtual Machine hosting appliance, hiding all that complexity and expose it via a web GUI or a Windows user interface.

There are always going to be lots of companies building tools on top of Xen. Even if you look at XenServer, there are all of these other companies building products on top of XenServer, like Egenera, Platform, Marathon. There is a very healthy eco-system of building stuff on top of other people’s stuf. I guess people are happy, because everybody is making money.”

06:27 Some analysts say Microsoft acquired Xensource by proxy, hinting at a future take-over of Citrix by Microsoft. What is your opinion on that topic?

“I truly do not know anything about that. I think if Microsoft was going to buy Citrix, it would have done so a long time ago. I think that Microsoft is a very close Citrix partner and that XenSource has worked with Microsoft as well. There are a number of projects on which we have worked together, such as defining some of the para-virtualization or OS-enlightment extensions to enable Xen-guests to run on Microsoft’s hypervisor when that ships and also vice-versa. We have always found Microsoft quite easy to deal with to be honest.”

(07:16) You get good support from Microsoft?

“Certainly all the people that we deal with are perfectly nice guys.”

(7:23) So let’s talk about the less perfectly nice guys & women. When I read articles on blogs and in the press, I feel that VMware is recently throwing some mud at Citrix and Xensource and especially the marketing department. They try to cast some doubt on your products and projects. What do you think about these marketing techniques?

“Well there has been a certain use of FUD-tactics and things like that. That is sort of a natural reaction. That is what marketing departments will go and do. We have good working relationships with some of the technical folks at VMware and we work together on the OVF virtual appliance format. I know that some of their engineers get pretty embarrassed about some of the stuff their marketing department does. VMware tries to position things which are Xen features or architectural implementations as ‘weaknesses’ against their product. Whereas they know they have teams working flat out to get and implement those same ‘weaknesses’ into their own product. That is just the way it is. Marketing departments go off and do that, but at the end of the day customers will hopefully get the right message and buy the right product.”

(08:45) At least it shows they take you seriously.

“I guess we should be flattered.”

View part 1 or part 2 of this interview.