IT 2.0 :  Next Generation IT infrastructures

The last day of March 2009 Intel officially unveiled its brand new Nehalem core architecture under the Xeon 5500 product name umbrella. There is not much to say about it other than it's impressive from a performance perspective. Just to give you a sense of what we are talking about the new product - only available for 2-socket servers today and with up to 4 cores per socket - has published many benchmark numbers that are either on par or slightly better than 4-socket Intel based servers with up to as many as 24 cores. One might wonder why a successful (and clever) company like Intel is going to cannibalize their highly profitable multi-socket market with a lower profitable product such as the 5xxx Xeon series. And I think the answer to this question is in one of the slides they used to present Nehalem at the launch event:

These numbers are impressive but I am pretty sure that if SUN and IBM marketing people would ever be able to read the small text at the bottom (which seems to be technically impossible) I am pretty sure they would come up with something to counter those numbers as they are obviously presented in a way that favors Intel; however I am not sure about this as I can't read the text myself so I don't know the assumptions behind those numbers. What it is important in this chart however is not the numbers (we know Nehalem has impressive performance per core) but it's the fact that Intel is now using Xeon to go after a 20+ Billion $ UNIX market. Up until now - and in the last 10 years - they would have been using Itanic (ehm... I mean Itanium... sorry for the typo) to go after the IBM Power or the SUN Sparc processors to get a slice of the Unix pie. This doesn't seem to be the case any longer. One might wonder where Itanium falls into all this: good question.

A bit of history on Itanium might help. Originally the Intel vision for the 64-bit Itanium was that it should have been the x86 32-bit follow-on product: the replacement for the Xeon brand basically. And they might have had a chance to succeed if AMD didn't come out with a much smarter evolution for x86 32-bit processors: in case you are wondering that would be an x86 64-bit architecture (namely AMD Opteron). When Intel understood they couldn't fight the Opteron with Itanium - since Opteron was 100% backward compatible with the Xeon software available whereas Itanium was basically not and would have required massive and painful applications porting - they decided to introduce the same "enhancements" to their Xeon processors. This was initially referred by Intel to as x86-32e: obviously they couldn't say Xeon was 64-bit as it would have overlapped too much with Itanium so they preferred to stay with the ridiculous definition of "32-bit Extended". This was the time where they tried to pitch Itanium as the only "native" 64-bit processor whereas the Xeon (as well as the Opteron obviously) were "just extensions to current 32-bit architectures". And this is when they shot themselves in the feet since they tried to play with the words (i.e. native sounds better than extended) but the only problem is that they forgot that, as far as IT is concerned,  native means you have to port the application whereas extended means it's compatible. So, for most of the customers, eventually extended sounded much (much!) better than native. And this is when Itanium started to see its decline in perception. I did a presentation at an IBM System x Symposium in France back in 2004 where I have shared these thoughts. Interestingly enough at that time we had an Itanium based System x box in our portfolio - the x455 - and I basically implied that Itanium (hence the x455) was at a dead-end and a useless product given the historical context we were facing. This is for example a chart that I used in 2004 to predict Windows on Itanium had no real place and didn't make any sense at all; it took a while but I think now MS think along the same lines:

Funny enough there was an Intel representative in the room that apparently didn't like these messages and he decided to escalate and complain about my pitch to my line all the way to the General Manager of the IBM Systems and Technology Group (that reported directly to Lou Gerstner - CEO of IBM at that time). I was never been officially involved in this complaint but the fact is that, later in the year, we dropped the x455. I like to think I gave a hint to the product marketing team on what to do but more likely what I said in the session might have been a blessing from the field about what product management was going to do anyway (and for very good business reasons). For your information I have posted the entire Power Point deck in the Files session of my site if you want to have a look. You can download it here.

To make a long story short Intel had nothing left to do than re-position Itanium as a high-end RISC replacement with the help of HP that, confident in its value and roadmap, decided to completely drop their own RISC offering - the HP PA-RISC processor - and jump onto the Intel Itanium processor as a strategic replacement. Intel tried to position Itanium as an open platform mentioning they had dozens of OEMs offering servers based on that processors but usually they forget to mention that the vast majority of the sales numbers they were seeing were coming from HP which is the only tier 1 server vendor today offering such a processor (IBM and Dell used to but they withdrew it and SUN never even attempted to).

As Xeon (and the AMD Opteron) became more and more enterprise-ready, the Itanium potential started to shrink even further. Up until now when Nehalem seems to be the last nail on the Itanium coffin. Consider also that the first Nehalem incarnation is a CPU model for 2-socket servers (Xeon 5xxx). This might leave the impression that Itanium can address a much larger window as it shines on highly scalable boxes. The truth is that this is the first product iteration based on the Nehalem core. Later in the year Intel will announce a multi-socket Nehalem based CPU - aka Nehalem EX - capable of scaling up to 8 sockets (Xeon 7xxx series). This CPU will feature 8 cores and Hyper-Threading thus providing execution support for 128 simultaneous threads (8 sockets x 8 core x 2 threads) in a single system image. Last but not least this new CPU will also feature additional enterprise functionalities such as MCA (Machine Control Architecture) which was one of the few things Intel used to position Itanium as "more enterprise" than Xeon. On paper a system like this could address the need for 99.9% of the customers' requirements. This statement obviously refers to performance but we obviously all know that performance is just one aspect of platform selection. This will obviously cause some adjustments in the server market shares and this goes back to the fact that apparently Intel is cannibalizing their current high-end market. Most likely what they have in mind, instead, is that they want to push the bar further and enter even more aggressively into the UNIX market with a more appealing and serious offering (than Itanium) like Xeon. The idea is: I will cannibalize a high-end x86 profitable market today which is worth a few B$ with a lower-end and less profitable product, because I want to use its big brother (Nehalem EX) to go after a 20B$ UNIX market. Since a picture is worth 1000 words this is what I am trying to say:

Note that I am not implying this is what I think it will happen. As I said performance is just a metric in platform selection. I am only speculating on the view that Intel has going forward. I am not ruling out completely (either) that this view has a point given what's going on and if this happens this will not only impact Itanium in the RISC space but other UNIX platforms as well.

Back to the Itanium discussion, last but not least it's worth mentioning that there is going to be a convergence in the Itanium Tukwila time frame (unsurprisingly delayed again) where you can drop this new CPU into a Nehalem standard socket (see the Update below). Intel has always pictured this flexibility as a mean to lower Itanium development costs and make it more flexible/cheap for customers and OEMs to move from Xeon to Itanium. The reality is that at the end of the day you end up having a common system, with the same components, with the same CPU socket. At that point you'll have the choice of installing either a cheap, super fast Nehalem processor with an unmatched flexibility of OS flavours and ISV applications... or installing a more expensive, somewhat slow Itanium Tukwila processor with an embarrassing flexibility of choice of OSes and ISV applications (at least compared to the Xeon family). I am pretty sure there are some HP execs regretting the port of HP-UX onto Itanium rather than having ported it onto the x86 architecture - if they knew 10 years ago what the x86 architecture would have looked like 10 years later.

It's well known that not only Itanium didn't bring any profit but its development costs have been impressive and they never got on par with slow sales. In a word Intel has lost tons of money on Itanium. Having this said there are obviously a number of issues that prevent Intel from dropping immediately the dead processor: for example contracts that they have signed with "these dozens of OEMs" - and one in particular which I won't mention (again) - that dropped their in-house developed CPU architecture for jumping on Itanium. They cannot just say "hey we are dropping Itanium" and leave these vendors in the mud (especially one). So I guess it's fair to say that, officially, Itanium is alive and healthy, obviously you can imagine what the reality is.

Massimo.

Update (10th June 2009): while Tukwila and Nehalem EX will share the same QPI bus the sockets of the two processors will continue to remain incompatible for the moment.

A few weeks ago I wrote a tutorial on how to deploy Hyper-V R2 on the IBM BladeCenter S where I demonstrated, among other things, how to LiveMigrate from one blade to another. I didn't spend too much time commenting on the implications this will have in the market. In this article, I'd like to comment on some of those potential implications.

Reading my piece you might have had the impression that I was "backing" Microsoft and putting Hyper-V R2 on the spotlight. That was not my intention: in fact the geek at the bottom of my heart just wanted to give it a try, as easy as it is. While I was pretty much happy with what I have seen, I was certainly not implying that Hyper-V R2 will be able to match VMware Enterprise technologies (both current and future). In fact, I don't honestly think that this is the case. Part of the misunderstanding is that, for some reason, this industry has grown with the stereotype that a virtualization product that is capable of moving a live workload from one server to another is to be considered enterprise-grade. VMotion has become the industry benchmark for being an enterprise product. I want to challenge this stereotype.

My article created a bit of confusion around this concept. "I saw your article. Are you saying that Microsoft is going to be on par with VMware?" is a common question I have heard a lot lately. I want to use this new article to give you the "other side of the coin" regarding these two important technologies Microsoft is going to bring to the market that are LiveMigrate and CSVs (Cluster Shared Volumes). While having these two capabilities in the new product will help Microsoft to overcome some limitations they have today for some deployment scenarios, this doesn't mean these features could be used in all scenarios (specifically Enterprise scenarios).

The devil is in the detail, so when you start digging a bit into the LiveMigration technology, for example, you can find that:

"..... On a given server running Hyper-V, only one live migration (to or from the server) can be in progress at a given time. For example, if you have a four-node cluster, up to two live migrations can occur simultaneously if each live migration involves different nodes....."

The full story is here: http://technet.microsoft.com/en-us/library/dd443539.aspx

This obviously is documentation that relates to an early beta of the product. But if they are going to stick with these limitations, it would be hard to imagine wide deployments in enterprise scenarios where you might require multiple live migration tasks going on cluster-wise at any point in time for resource optimization reasons. So assuming Microsoft (or Citrix with their new Essentials for Hyper-V package) will come out with some sort of DRS-like product in the R2 timeframe, they might not have the underlying infrastructure ready to leverage these add-on tools.

The same goes for Cluster Shared Volumes: the devil is always in the details. If you have read my previous article you might have had the impression that CSV will deliver pretty much what VMFS delivers today. Well, apparently yes, but again, if you dig a bit into the details you will find out some limitations that might not be relevant for small deployments but might be show-stoppers for enterprise deployments.

  

At the time of writing, these slides were publicly available at this link. Kudos to Microsoft for not hiding these details and for letting the people know about the limitations.

While it appears at first that CSVs are a "transparent" technology, the reality is that as soon as you start pushing the envelope, they are not. How many enterprise IT organizations today leverage storage replication technologies to implement Disaster/Recovery scenarios? Based on my experience I would say many of them. Hyper-V R2 with CSVs will break this common implementation pattern if they won't be able to overcome this limitation. A pattern that I would imagine all these enterprise customers want to continue to leverage and something that is not just bound to current VMware deployments as it's a technique that is being leveraged by UNIX and Mainframe deployments as well to achieve High Availability and Disaster Recovery.

These are just two examples. As I said, supporting techniques that allow a live workload to fly from a physical server to another is just one aspect, but probably not even the most important. The fact that you have a small Cessna - and so you can technically fly - it doesn't mean it's the most optimal, secure and comfortable means of transportation to go from Milan to New York. For that you want to fly on a 767 (and in business class, if possible!). Of course there are a lot of Cessnas around as they fit a part of the market.

On the other hand, as I said in my previous article, Microsoft has a tremendous asset: they are making (almost) everything available for free. Which leads to at least a couple interesting comments.

Does the price discussion really matter anyway?

The first comment is: does the price discussion really matter anyway? The Microsoft pricing strategy is so that when you have properly licensed your Windows guests (typically via either Windows Server Enterprise or Datacenter SKUs), your underneath Microsoft Hyper-V virtual infrastructure is already licensed by definition. And this is true today. Suppose you have 50 Windows guests to deploy on four 2-socket servers for example. Most likely the cheaper way to license these 50 guests is via the Windows Server 2008 Datacenter SKU which is licensed per physical socket and provides unlimited number of guests. If you do so, it doesn't really matter whether you want to use Hyper-V 2008 Server or a full-GUI Windows 2008 Server w/ Hyper-V or a GUI-less Core Windows 2008 Server w/Hyper-V as your parent partition. You have the right to use everything you want for free including the Failover technology (Microsoft Cluster Server). This excludes the MS Virtual Machine Manager but this won't change in the Hyper-V R2 time frame. So this claim that with Hyper-V R2 they will have more stuff for free is a bit misleading in my opinion in the sense that they already effectively provide many things today (not just the Hyper-V Server SKU) for free.

There is a caveat to this, though, and it boils down to how customers are going to license the Windows guests. The analysis above assumes that customers are going to buy brand new licenses for their new deployment (because they had OEM Windows licenses on their old physical servers that could not be repurposed, for example). If the customer has Windows licenses that they can repurpose on the new virtual infrastructure, then the discussion on the cost of the virtual infrastructure itself is no longer trivial. And, yes, there will be a big bonus in this regard during the R2 timeframe - as the free Hyper-V Server R2 version will have more features and fewer limitations than the current free version. The pricing discussion can get very complicated, as mentioned in my blog a few months ago. It would be interesting to see some statistics on how customers have currently licensed their legacy physical servers.

Last but not least, I am assuming that these customers are using Windows guests on their Hyper-V infrastructure. While Microsoft supports a limited number of Linux distributions (today SUSE, but they announced future support for Red Hat, too), I don't see too many Linux-only customers leveraging Hyper-V for their virtual infrastructure deployments.

Clearly the Microsoft virtualization strategy is different than the VMware virtualization strategy

The second comment regarding the (virtual) price war is this: clearly the Microsoft virtualization strategy is different than the VMware virtualization strategy. And the pricing strategy reflects that. I wrote another article on this topic which I invite you to read. I am attaching hereafter the picture for your convenience because I want to use it to back my point.

 

In a nutshell, Microsoft makes money out of the red part whereas VMware makes money out of the blue part. Microsoft is probably going to stick with their "Virtualization is a value item of the OS" strategy for the time to come if the pricing schema for Hyper-V R2 (due early next year) is what they are pitching today. Basically what they are doing growing the blue part, and giving it away for free. The only way they can sustain this is by continuing to make money on the red part. This has at least a couple of implications that are worth underlining:

1. Their Linux strategy is pretty much opportunistic (well, it's obvious and totally expected after all - it's a dumb statement) in the sense they want to give customers with a "few Linux servers here and there" the possibility to leverage the Hyper-V infrastructure these customers are using for the majority of the (Windows) VMs. Even though a Linux shop (probably) would not want to use Hyper-V for technical (or religious) reasons, it wouldn't even make sense for Microsoft to go down that path because they would need to make the blue part technologically compelling on its own while giving it away for free. There would be no revenue stream for Microsoft in such a scenario so probably not worth the effort for them.

2. Microsoft will not have a business interest in making the blue part grow too much as long as they are going to give it away for free. This means that they won't be able to afford to be so aggressive in the Virtual Appliance space because the JEOS concept is pretty dangerous to their current business model as you may detect from the picture (the smaller the red part is the less leverage they have). Unless they radically change their software licensing model - which I wouldn't rule out- I don't see how they could sustain an aggressive move toward this JEOS concept. Consider also that the smaller the red part is, the easier it could be to migrate to a different OS for the ISV. This is a generic statement obviously and might not be applicable to specific situations.

All in all what Microsoft is doing is interesting and it will benefit customers because it will keep VMware honest in what they are doing - in terms of both technology and pricing. My speculation is that this is going to be a two-horse race in the long run between VMware and a virtual agglomerate comprised of Microsoft and its historical partner Citrix. There are concerns and rumors in the industry - admittedly, I am personally backing them - that Citrix has sort of lost interest in battling at the XenServer level, which is now being distributed for free. Some people are speculating that Citrix is shifting their strategy to expand and provide value on top of someone else's basic virtualization offering (namely Microsoft Hyper-V) and losing focus on their own commodity hypervisor and management offerings (XenServer). Similar to what they are already doing with XenApp expanding the core Microsoft Terminal Services technology.

There is no question that the aggressive pricing move from Microsoft in the R2 time frame will garner some reaction from VMware. I don't have any insight but I wouldn't be totally surprised if VMware was going to provide VMotion either for free or in one of the less prestigious future vSphere SKUs. There are enough technology deltas, on top of VMotion, that will differentiate VMware from Microsoft (especially for enterprise deployments) that will allow the guys in Palo Alto to continue to charge premium prices if they want to.

However, I think that VMware will be at a fork sooner or later: they could either continue to charge a premium for their unmatched features to fill a need some of the Enterprise customers have (and that no one in this industry can or will match), or they could substantially lower their prices to appeal many more customers - especially those that can't afford their technologies. The theory is that you could earn $1,000 either charging 1,000 customers $1, or charging 100 customers $10. This always holds true unless you figure out a way to charge 900 customers $1 and 100 customers $10. Their Acceleration Kits are an attempt to achieve that  value proposition, but what Microsoft is doing in the R2 time frame might require a revisiting of the current VMware portfolio layout (which I am sure is in VMware plans). Of course, we need to remember R2 is still about a year away so VMware has some time to think about this.

Massimo.

My good friend at Microsoft, Giorgio Malusardi, noticed my post "Enterprise Virtualization in a Box" which was essentially an example of how to create a BladeCenter-contained VMware-enabled data center in a box (including servers, storage and networking). Giorgio challenged me with the task to create something similar using the Hyper-V Server R2 Beta that has just been announced. And I accepted the challenge!

This tutorial is going to document the setup of the environment based on what I have seen and I have done. I will share my point of view of what's going on and the implication this will or might have in the x86 market in another piece. 

Microsoft Virtualization Background

For those of you that are missing the Microsoft basics it would be beneficial to set the stage. Right now, Microsoft is shipping the first version of their hypervisor - Hyper-V - by means of two different channels. The first one is as a component (or role) of their Microsoft Windows Server 2008 products. You can enable or disable this role in either a normal (GUI-based) Windows Server 2008 install, or a core (GUI-less) Windows Server 2008 install. Obviously, in order to get Hyper-V, you need to buy a Windows Server 2008 SKU (Hyper-V is included in any 64-bit x86 version of the Standard, Enterprise and Datacenter SKUs). The license rights for guests and included features - such as Failover Clustering technology - are determined by which SKU is purchased.

The second channel is as a free download from the Microsoft web site in a package called Microsoft Hyper-V Server 2008. In a nutshell this is basically a scaled-down version of Windows Server 2008 with the following restrictions and peculiarities:

• It is a core install only (i.e. GUI-less as the only option)

• The only role that it supports - which is enabled by default - is Hyper-V (for example, you can't enable the Failover Clustering role)

• It doesn't include any license for Windows guest OS'es

• It does have a number of artificial limitations in terms of number of CPUs and amount of system memory supported.

That's what's available as of today. However, Microsoft recently announced the availability of the Beta version of Windows Server 2008 R2 and Hyper-V Server 2008 R2. Both these products will ship the second generation of the Hyper-V hypervisor and are currently scheduled to ship in about a year from now (roughly). With this Beta, Microsoft announced new features and new restrictions for the free package. The following table is a summary of the features in the current and future offerings:

 

* Cluster Shared Volumes is a technology currently in Beta and will ship along with the second generation of Hyper-V. It allows to use the NTFS file system as if it was a "cluster file system" (ala VMFS so to speak). See below in the document for more information on the CSV technology.

Those of you familiar with the Microsoft Virtualization technology will notice that the Windows Server 2008 R2 SKUs will have similar restrictions and limitations compared to the current releases. This statement obviously doesn't take into account new features introduced with the second generation of the hypervisor (such as Live Migration, for example). As you may have noticed, the biggest delta both in terms of new features and artificial limitations is between the currently shipping Hyper-V Server 2008 (first column from the left) and the future Hyper-V Server 2008 R2 (second column from the left). Among many differences, it's specifically worth to note that the new (free!) product will support:

• 8 sockets (vs. current artificially limited 4)

• 1TB of memory (vs. current artificially limited 32GB)

• Quick and Live Migration (vs. nothing)

• Failover Clustering (vs. nothing)

• Cluster Shared Volumes (vs. nothing)

 

The Hyper-V Server R2 Based Self-Contained Data Center

Back on track. As I said, the challenge was to replicate the VMware-based setup we have done on the BladeCenter S. We have used the very same hardware setup we have used for the VMware test. While we wanted to test the Hyper-V Server R2 Beta it must be noticed that the currently shipping Hyper-V solution works as well on the BladeCenter S today. This is a (generic) picture of the BladeCenter S chassis:

For this proof of concept, I decided to look at the things from the following perspective:

• I wanted to focus on the Hyper-V Server R2 free product (and not on the general purpose Windows Server 2008 R2 w/ Hyper-V role enabled)

• I wanted to focus on new technologies that will be shipping in the R2 timeframe. This includes CSV, Failover Clustering and Live Migration

• I wanted to focus on what you could do with the future Microsoft free offering. This includes the standard free tools to manage the environment and obviously doesn't include the fee-based products such as Virtual Machine Manager (the current version wouldn't support Hyper-V Server R2 anyway and there is not a "sister Beta version" of VMM to test with the Hyper-V R2 Beta bits).

All this being said we can "replay" what I have done.

 

Hyper-V Server R2 Nodes Setup

First, I started installing Windows Hyper-V Server R2 on the two local disks of the two blades in the chassis. This is a picture taken from the Management Module of the BladeCenter S during the setup (remote attended install):

I could have set up the basic OS on the shared storage as well as dedicating a small LUN to each of the two blades but I remember there was a registry tweak to apply in the Windows 2003 timeframe to allow a single shared SAS/FC to handle both the C:\ drive as well as the shared storage in a MSCS scenario. I didn't want to get into that level of complexity, especially as it was not one of the main goals I had with this Proof of Concept. Enough to say that I am sure you could get rid of the local disks if you really want to.

The setup doesn't really ask too many things. Actually nothing. At the next reboot you are asked to change the Administrator password and off you go. This is what you get on a Hyper-V Server R2 Beta local console:

Through the Hyper-V Configuration panel (blue window), I did the following:

1. Changed the default Host Name (into HVR2NODO1 and HVR2NODO2)

2. Restarted server to apply the computer name settings

3. Changed the IP to static addresses (192.168.88.131/132)

4. Enabled RDP support

5. Configured Remote Management to allow WinRM and relax Firewall settings

6. Enabled an extra firewall setting (through the command Netsh advfirewall firewall set rule group=“Remote Volume management” new enable=yes) for managing the disks through a remote MMC snap-in

7. Joined the domain (Windows 2008 R2 Domain created on a separate server on the network)

8. Added the domain Administrator to the local Administrators group (option 4 of the Hyper-V Configuration tool).
    

At this point - before enabling Failover Clustering support - I configured both blades to access two shared LUNs created with the IBM Storage Configuration Manager, which is the tool you can use to configure the BladeCenter S integrated storage. This picture shows that a Quorum LUN (10GB) and a CSV LUN (100GB) have been assigned to both blades in the chassis.

A restart of both blades allowed the domain change to take effect as well as the disks to be recognized by the two Hyper-V Server R2 instances (alternatively, a disk rescan would do this job).

Because of the fully redundant fabric architecture of the BladeCenter S, the two disks we have just configured (Quorum and CSV1) are seen twice by the hypervisor OS because of the dual path that each blade has to get to the disks (this is, by the way, the big plus of this chassis with the integrated storage). A multipath I/O software needs to be installed on the Hyper-V hosts to manage the disks properly. This is done by first enabling Hyper-V-based MPIO support which is not installed by default. The command "oclist" displays all features that have been enabled/disabled on the host as you can see from the picture below:

On one of the two hosts, I manually enabled base Microsoft MPIO support (via the command "start /w ocsetup MultipathIo"), but this is not enough. I had to install storage specific multipath software which interacts with the base Microsoft MPIO code. In IBM terms this is called IBM Subsystem Device Driver and can be downloaded off the external website. At the time of this writing, the package is located at this link and it's called the "SDDDSM Package for RSSM" (SDDDSM= Subsystem Device Driver Device Specific Module; RSSM=Raid SAS Switch Module). It's interesting to notice that the package in subject has a typical Windows setup, so I was wondering how it could be installed on a GUI-less system. Well, launching the setup.exe did the job, as you can see in the following pictures.

 

First impression was that this was not really a GUI-less system, but rather a standard Windows system where explorer.exe was disabled. Well, never mind....

After the reboot the system was up and running again, and the hypervisor correctly reported only two disks being assigned to the blade (the 68GB disk is the local hard drive whereas the 100GB and the 10GB are the two LUNs I created with the Storage Configuration Manager utility).

On the second blade we found out right away that installing the IBM SDD software automatically enabled Windows base MPIO support (if it doesn't just use the command above to enable it).

At this point we enabled the Failover Clustering feature on both hosts via option #8 of the Hyper-V Configuration window. This enables the Microsoft Cluster Server code on the two hosts. The picture below shows what happens on the console when you enable this feature. The Cluster itself will be configured later.

This is pretty much it for the Hyper-V hosts setup. This concludes the configuration of the base support that needs to be done on the Hyper-V Server Configuration console. From now on we can do pretty much everything from the Microsoft remote tools.

 

Hyper-V Server R2 Nodes Configuration from a Remote Workstation

We can switch focus to a Windows 2008 R2 Server that we previously installed and configured to be a Domain Controller for our test bed. Remote administration of the Hyper-V hosts could either be accomplished from this host (after enabling some remote administrative tools that are disabled by default) or from a Vista / Windows7 workstation using the latest RSAT tools available from the Microsoft web site. These tools include advanced Remote Administration MMC Snap-Ins that don't ship with the base client OS and allow to do enhanced tasks such as Live Migration. The latest release of these tools (in beta) can be downloaded here.

If you use the workstation it must be in the same domain you joined the HyperVR2 hosts to. If it is not in the same domain, extra configuration steps on the Hyper-V servers are required to relax cross-domain security restrictions.
Since one of the purposes of this test was to demonstrate how you can remotely manage advanced hypervisor features using free tools, we have created an MMC configuration (that we called "MasterMMC") which includes the following Snap-Ins:

• Remote Disk Management

• Failover Cluster Manager

• Hyper-V Manager.

I have used the Remote Disk Management tool to configure partitions and file systems on the two shared disks on both blades: I have assigned the Quorum LUN the Q: letter and the CSV1 LUN the X: letter on both nodes to prepare for cluster enablement. Initially I had a hard time getting to the Hyper-V nodes via this applet. I eventually managed to get to a stable state where I could manage the disks, but I have had many connection issues ("RPC Server unavailable") that I couldn't nail down to a particular problem. Firewall issues as well as bugs in the code (which didn't refresh the pane properly for which I had to close and re-open the MasterMMC) might be potential causes.

The Hyper-V Manager Snap-In was more straightforward. The only thing I have done here is assigning the second Gigabit adapter on the blade to a VirtualSwitch (called VMs in the screenshot below) that I have defined on both Hyper-V nodes. The first NIC (which I have configured with a static IP address at the beginning of the setup) remains assigned/dedicated to the parent partition.

This is the "network" (aka VirtualSwitch) to which you will connect the guests to get physical network access.

Notice that the BladeCenter S supports blades with up to 4 NICs configured. For this test only two NICs have been configured on each blade. Remember, Hyper-V currently does not allow NIC teaming at the hypervisor level (i.e. assigning more NICs to the same VirtualSwitch). Microsoft advises to use third-party NIC software to create bonds of network adapters and assign the resulting "bonded NIC" to the VirtualSwitch. It's not clear whether Hyper-V R2 is going to change this when they ship the gold code.

The next step is to configure the cluster across the nodes. This is not really Hyper-V Server specific, as the procedure is pretty similar to what you would do on a Windows 2008 Enterprise Server. It involves validating the hardware setup first with the built-in utility and then configuring the cluster properties (clustername, IP address etc).

Next, I enabled Cluster Shared Volumes. Those of you that are familiar with Hyper-V and Failover Clustering know that in order to manage a Guest as a single entity (i.e. independently "Quick Migrating" a VM from one host to another) the VM needs to be created on a dedicated shared LUN. This is, by the way, the configuration Microsoft usually advises. This has a number of implications in that you could easily run out of drive letters in the cluster (this can, however, be by-passed using specific mounting techniques), but more importantly it introduces a management overhead: you need to create a LUN for each VM you need to deploy, rather than leveraging a BIG shared LUN cluster-wise (like VMware VMFS allows you to do). That is what CSVs are all about: they provide a "cluster file system"-like environment where you can run a number of different guests on different hosts pointing to the same shared LUN. In fact, it's not by chance that I have assigned the blades a 10GB disk to be used as a dedicated Quorum, as well as a unique 100GB CSV1 LUN to be used concurrently as a shared repository to host multiple VMs. This is obviously a new and big benefit since the current Microsoft Cluster Server architecture is such that if a node owns and can access a LUN the other host in the cluster is inhibited from accessing it (at least until the group containing the LUN fails and the cluster changes its ownership).

The picture below shows the disclaimer about CSV: they can only be used to host virtual machines in a Hyper-V R2 environment! This means they can't be used in a general purpose Windows Server 2008 Microsoft Failover Clustering scenario.

The cluster configuration wizard asks me which volumes I want to enable: CSV1 is the only remaining partition I have (the Q: drive has already been used for the Quorum):

Once the CSV has been enabled, on each cluster node a new directory structure appears. The default is "C:\ClusterStorage\Volume1"

This is a "virtual pointer" that refers to the CSV1 LUN and it shows up on each of the two blades and describes a sort of common/shared name space that both blades can access at the same time. This concept applies to virtual machines only and the usage of CSV cannot be extended to a general purpose cluster file system at the moment.

Now that we have a cluster set up and a CSV volume available, we are going to create a virtual machine. We point to the Hyper-V Manager Snap-In in the MasterMMC window and we configure the VM to be hosted on the CSVs explicitly choosing the common local name space that identifies the CSV on the Storage Area Network:

At first it seems to be odd to create a to-be-clustered virtual machine on a "C:" drive, but that's the way it works. Obviously the VM files won't be created on the local drive on the blade because, as I said, that path represents a location that is actually on the SAN. This is how our MasterMMC looks like in the end once we have done all this:

So far we have only created the VM. It's not yet clustered, as there is no integration between the Hyper-V hosts and the Failover Clustering applet, using the free management tools. Microsoft Virtual Machine Manager is supposed to provide this integrated view and operations, but as I said at the beginning, the currently shipping VMM version doesn't manage Hyper-V R2 Beta hosts yet. Besides, it would be beyond the scope of this document anyway. So in order to clusterize the VM we have to explicitly and manually declare this VM as a clustered resource. The steps are similar to how you would configure any cluster resource; just make sure you select "Virtual Machine" as a resource type and then you are presented with a list of VMs that are running on the cluster hosts (i.e. both Hyper-V R2 Beta servers). Notice that the virtual machine needs to be powered off to be clusterized (otherwise the wizard will fail).

Once we have configured the resource we can bring the virtual machine on-line:

The resource (virtual machine) is now online and it's running on the second Hyper-V R2 node (HVR2NODO2) as you can see from the picture below:

At this point you can invoke from the Failover Clustering interface a "Live Migration" of the resource as you can see below:

And the virtual machine will start the live migration onto the other host:

During my test I have been able to successfully move the virtual machine from one node to the other with basically no downtime except for a ping or two:

Consider the networking configuration might not be optimal, and we will have to see what Microsoft will suggest in terms of network subsystem setup in the context of Live Migrating a virtual machine. Having said this, loosing one or two pings is usually something most web and client/server applications would be able to handle, and it's not too much different from the experience you would have using alternative live migration technologies from other vendors such as VMware, Citrix, VirtualIron etc.

The last test of this proof of concept is to create another virtual machine and demonstrate that they could run simultaneously on the two Hyper-V R2 Beta hosts while insisting on a common shared LUN through the CSV technology. These are the screenshots of the two virtual machines running on different hosts but insisting on the same repository which is the CSV1 volume mapped on both hosts as "C:\ClusterStorage\Volume1":

 

There is one pretty interesting thing in these, if you noticed. Despite the fact that both nodes can access the CSV at the very same time (otherwise they couldn't simultaneously run two virtual machines hosted on the same volume), the actual LUN is, at any point in time, "officially owned" by one of the two nodes (in this case the owner of the LUN is always HVR2NODO2). I must admit I have to dig more into the CSVs but they seem to be arbitrated and controlled by one node at a time. My assumption is the cluster node that is NOT the owner of the LUN would not use the owner of the LUN as a proxy to get there because this would hurt substantially the disk access performance (i.e. one node has direct access while the other node has a pass-through access through the owner of the LUN - not a viable scenario). Somehow the other node (i.e. HVR2NODO1) has direct access to the LUN performance-wise but it also must coordinate access rights with the official owner of the LUN itself (that is HVR2NODO2).

In a scenario like this it would be interesting to understand what happens when the node that is the owner of the CSV crashes.

To recap, this is the summary of my current setup:

                                             Running on                         CSV Owner                              

VirtualMachine ( )                   HVR2NODO2                      HVR2NODO2

VirtualMachine (2)                  HVR2NODO1                      HVR2NODO2

In a cluster file system environment, if HVR2NODO2 fails, VirtualMachine(2) would continue to run on the other node (HVR2NODO1) without any interruption and VirtualMachine( ) would go off-line to restart on the same surviving node (HVR2NODO1).

So I turned off blade #2 in the chassis (which is HVR2NODO2) via the remote BladeCenter S Management Module (MM):

VirtualMachine(2) didn't experience any issue both from either a ping perspective or a Failover Cluster Manager notification. This would lead me to think that CSV ownership would change transparently without any service interruption. This was somewhat expected and the only point of concern was the ownership of the CSV (which apparently can be managed in a smart way). However, the other virtual machine experienced downtime. This was expected as well, since VirtualMachine( ) was running on HVR2NODO2 which was turned off "in the hard way" so the failover algorithms had to kick in to bring it back on-line on the surviving node (HVR2NODO1) with a standard boot-up procedure.

Notice that the ping window first loses the link, then it starts to get a host destination unreachable message from the local IP address (192.168.88.133 is the host from which I am pinging). Eventually it starts to ping the guest again once it's brought back on-line.

Preliminary Conclusions and Impressions

As I said at the beginning, I will write another piece on what I think the implications of these technologies will be in the market. From what I have seen so far, the Hyper-V R2 platform seems to be pretty stable (once I got passed some weird issues with the Remote Disk Management stuff). Let's not forget that we will not see these technologies before year end 2009 or the beginning of 2010. This is the common speculation in the industry, anyway. While this will allow plenty of time for Microsoft to fix these problems, the fact that these are still one year away will give VMware some time to think about their main competitor.... although I am sure all this is already on their radar in Palo Alto.

There are a number of aspects in the Microsoft technologies that I think are a long way from catching up with what VMware is doing. VMware had the advantage of starting to develop a true virtualization platform from a blank sheet. Microsoft, on the other hand, has a legacy of technologies, so virtualization for Microsoft seems more hammered-in than anything else. An example is the fact that when you create a Virtual Machine from the Hyper-V Manager, the default location is "C:\ProgramData\Microsoft\Windows\Hyper-V", which is not what I would define as a proper default location for hosting enterprise workloads (in fact, it looks more like a Microsoft Office document default location). This might sound simple, but it tells you a lot about the heritage Microsoft wants and needs to protect.

That's pretty much it for the negative part. As far as the positive aspects are concerned, everything you have seen here (except the BladeCenter S and the Windows guests!) is all software that is free of charge. And this is not a trivial aspect or something to overlook.

Massimo.

In this post I am going to talk about a specific piece of hardware technology that is intercepting a specific virtualization industry trend. This piece of technology is called BladeCenter S. Those of you that have been reading my blog know I don't usually talk about IBM specific stuff (I work for IBM) but this time I felt like the infringement of the law was worth it. Believe me or not I would have posted this anyway.

Before we get into the specific of the technology let me take a step back and briefly touch on the industry trend I was referring to. This is going to be basic stuff for most of the virtualization experts out there plus these concepts are not new and I have written/talked about those in the past. Having this said sometimes it's good to pause for a second and try to summarize what is happening in this industry. Up until the late nineties (almost) every data center looked something like this:

Very inflexible and vertical silos. Each silo was comprised of the following building blocks:

• A server

• A local disk subsystem (aka DAS - Direct Attached Storage)

• An operating system

• An application

Do you have 100 application services? Deploy 100 of these independent silos! Have you ever heard virtualization (true or appointed) experts talking about how bad life was those days? Look at the picture... and you can imagine how life was. I can tell you: it was very bad (compared to what we have today obviously, at that time it was... OK).

At the beginning of the 21st century we have started to see the very first form of "visible" virtualization of an x86 IT infrastructure. I am using the world "visible" because someone might argue that the concept of virtualization was already included in the OS under the form of memory virtualization (physical memory Vs virtual memory etc ; I am not interested in these academic discussions and I am not interested in determining where virtualization first appeared in the x86 ecosystem (we can stay here for days without getting to any useful outcome). I want to focus more on tangible things that end-users/human beings (not IT geeks) understand and can appreciate. Having defined the context, the first form of "visible" virtualization of an x86 IT infrastructure was the storage and particularly the consolidation of all Direct Attached Storage into a single pool of storage resources called SAN (Storage Area Network). And since my mantra is that a picture is worth 1000 words, here it is how a common x86 IT infrastructure looked like at the beginning of this century:

Note: If you ask 100 storage specialists nowadays what storage virtualization is you might very well get 100 responses (perhaps more?) ranging from "Raid 0 is the basic form of storage virtualization" all the way to "a storage grid (whatever that is) is the only form of storage virtualization". I am using here the word virtualization in the context of storage to describe the high level practice of decoupling the disk subsystem from the servers and locate it into a common resource pool.

Back to the basic this is what customers have been doing for the last 10 years or so: getting rid of this locally attached / inefficient / inflexible disk subsystem and move (almost) all the disk spindles into a central repository that is the so called Storage Server (the physical data repository attached to the SAN). The very first advantage that this has brought to customers is a more efficient and flexible way to use the storage space; someone might refer to this as Storage Consolidation. On the other hand shared consolidated storage brought in (as a bonus I would say) a brand new architecture that allowed customers to do things that were not simply possible before. One example for all is High Availability clusters: in the good old days of DAS (and the inflexible silos described at the beginning) your application data would most likely be hold physically on the same server that was running the application. Should that server fail you couldn't access any longer your data (unless you restore them from a backup); with SAN shared storage this changed as you can now "attach on the fly" the same set of data to another server and restart the application from there while being consistent in terms of data persistency. Microsoft Cluster Server, anyone?

Well time goes by and right now storage virtualization is no longer the hot topic (I guess everyone recognizes it as more of a prerequisite to run an efficient IT). The buzz word today is server virtualization and, if you think about it, it's the natural progression of what we have seen happening in the past: it's about taking the silo apart and move additional stuff below the virtualization bar. We have done that with storage, who's next? Did I ever say a picture is worth 1000 worth?

This is where we are today basically. VMware pioneered this concept some 10 years ago and there is now a string of companies that have realized the benefits of this and are working hard to deliver products to implement this idea. I started working on server virtualization some 8 years ago and at that time it was all about server consolidation (i.e. how many servers do you have? 100? we can bring them down to 5 etc . The more I was working on it the more I understood that we were only scratching the surface of the potentials. Today server consolidation is still a huge advantage for those customers virtualizing but it's clearly only one of the many advantage line items. As it was for storage virtualization we started with the consolidation concept to find out that there were many other hidden and indirect advantages as a bonus of doing that. One example for all is that, as you virtualize your Windows or Linux systems, it becomes far easier to create a Disaster/Recovery plan for your x86 IT infrastructure.

Last but not least the server virtualization trend is intimately associated to the storage virtualization (i.e. SAN) trend for two key reasons:

1. the standard server virtualization best practices require shared storage to exploit all the benefits

2. server virtualization is allowing customers to get rid (completely) of local attached storage. While data has been historically moved to a shared repository (SAN) the standard "2 x Raid1 drives pair" remained a (negative) legacy of the x86 deployments. The latest trends (that are embedded hypervisors on flash disks and/or PXE boot techniques for the hypervisors) will help getting rid completely of all the local server spindles for good!

So why am I so excited about the BladeCenter S you might wonder? Well the BladeCenter S maps exactly the industry trend I have described above. Instead of going out for shopping and cabling together all these elements (servers, SANs, etc) BladeCenter S is a single package that contains them all: servers, storage and network! Enterprise Virtualization In-a-box! Or a data-center-in-a-box if you will!

What you see here is basically the physical view/package of the de-facto-standard hardware architecture to support virtual environments. The key point I am trying to outline here is that the disks you see integrated into the chassis are really connected to a true fully redundant internal SAN comprised of 2 x SAS redundant RAIDed switches. It essentially maps the standard servers to storage architecture blue-prints we have been using in the last few years to implement shared storage virtualized deployments. The following picture, for example, is an extract from the standard VMware SAN configuration guide and it illustrates this standard blue-print (which is mapped into the BladeCenter S internal architecture):

Notice that the only slight difference is that the SAS switches integrated into the BladeCenter S deliver both switch as well as SP functionalities.

It might perhaps help sharing with you some more documentation I have been working on and that we presented at the local VMware Virtualization Forum that took place in Milan a few days ago. The following picture describes the internal architecture of the BladeCenter S in further details:

 

Notice how the servers-storage connections are similar in concept to those in the standard VMware blueprint (but not limited to VMware deployments though) attached above. Each blade is equipped with a dual-port SAS HBA which in turn connects to 2 x SAS RAIDed switches which control the disks. For those of you familiar with the IBM storage products family this is very similar to what happens when you connect ESX servers to an external DS3200 SAS Storage Server configured with dual controllers. Since in the last few months I have been talking to customers and partners that were pretty confused about what this really is and how it compares to other implementations available in the industry I did want to outline what other blade vendors are doing to underline the differences:

 

 

While from a physical standpoint it might look pretty similar (i.e. "a chassis with a bunch of blades and a bunch of disks") if you dig into the internals it's of course completely different. The other option outlined in the picture above involves dedicating a single blade (hence a Single Point Of Failure) with Windows Server 2003 Storage Server and a bunch of disks attached to it. The Windows instance running on the Storage Blade controls the disks and exposes them onto the internal Ethernet network via NFS/iSCSI protocols. This is how other blades in the chassis can "share" those disks. There are, obviously, fundamental differences between having a multi-purpose Windows blade sharing disks over the network compared to using a standard and fully redundant SAN approach comprised of a dedicated couple of purpose designed SAS RAID switches that control the disks and map those disks to compute nodes (i.e. the blades dedicated to the virtual infrastructure). The following picture reminds the physical layout of the BladeCenter S with the integrated SAN.

 

 

On the left hand side you can see the front of the chassis where the disks (we had 4 of them in our demo on-site) and the blades (2 x HS21XM in our setup) are installed. On the right hand side the rear view of the BC S chassis shows the 2 x Ethernet switches (that can support up to 4 Ethernet connections from each of the blades) and 2 x SAS RAIDed switches (that control the disks on the front of the chassis and are connected to the blades by means of the SAS daughter cards).

Another interesting point I wanted to outline via this setup is that the BladeCenter S is really meant to be a self-contained data center. This doesn't only include the standard User Workloads (i.e. the guests that are going to support the customer own environment such as Active Directory, Databases, Web Servers, Application Servers etc) but it also includes all the additional services that are required to configure, monitor and maintain the data center (in a box). Examples of these System Services include the vCenter service (red rectangle in the figure above) which can be installed on top of the virtual infrastructure as well as what I refer to as the HW Management service which is the suite of software products that are used to manage the hardware and its configuration (the yellow rectangle in the figure above - it might include things like IBM Systems Director, IBM Storage Configuration Manager etc). The logical view shows these two services (vCenter and HW Management) as external entities that map respectively the ESX hosts comprising the virtual infrastructure and the Management Module (MM for short) that is the heart of the BladeCenter chassis. There is no reason though for which these services need to be installed physically outside of the BladeCenter "domain". A forward-looking take of these services is to consider them a sort of System Partitions that run side by side with the end-user workloads. These System Services, as of today, need to be installed manually but ideally in the future they could potentially be distributed as Virtual Appliances (yes Virtual Appliances is my obsession, sorry) for a more streamlined and fast deployment.

In the next few screenshot I'd like to give you a high-level feeling of what happens when you connect to the HW Management service to configure the hardware components (the shared storage in this case). For this setup I have only installed the IBM Storage Configuration Manager in that HW Management System Partition.

 

First you connect, via web, to the SCM service. One of the main screen summarizes the actual internal hardware storage configuration which is a RAID subsystem comprised of 2 x SAS switches:

 

Next is the physical view of the chassis. As you can see we have 4 x physical disks plugged into the front of the chassis and 2 x physical SAS switches in the back of the chassis (the two additional devices you notice in the front are the SAS controller caches). A maximum of 12 physical disks can be installed:

 

The following view details the characteristics of the physical hard disks:

 

Next we create a Storage Pool (aka Array) comprised  of these 4 physical drives. This is a very basic configuration where we designate one of the disk as a global hot spare and three of the disks as a Raid 5 Storage Pool. Total available capacity is 2 disks (1 is used for parity in a RAID 5 array). Notice that the space available is basically 0 because I have already created LUNs out of this array (see next):

 

These are the two Logical Units (aka LUNs) that I have created using the Storage Pool described above. One is 90GB and the other one is 43GB in capacity:

 

The following view lists the discovered SAS daughter cards (hence the corresponding blades) on the SAS fabric. Notice that each blade has two ports for redundancy and each port has its own SAS WWN. This is not any different from a standard FC configuration for those of you used to Storage Area Networks:

 

This is how I have mapped Servers to LUNs. On the left hand side I have listed both blades whereas on the right hand side I have listed both LUNs I have created. Doing so I allowed both blades to share both LUNs. There is no particular reason for which I have created 2 LUNs. I could have created 1 or 3 or 4 if I wanted/needed to and I would have been able to share them with both blades:

 

So far we have been working against the HW Management to configure the hardware (this example is limited to configuring the shared storage). Now we can switch gear and we can connect to the other System Partition to manage the virtual infrastructure software. In this case we will connect to the vCenter service to configure our VMware infrastructure. Notice that, although I have been using a beta version of the next VMware virtual infrastructure product, everything you will see here can be done with the latest VI3 version available today.

The following screenshot outlines the overall configuration of our data-center-in-a-box. As you can see there are 2 blades equipped with ESX and they belong to a cluster. On these blades we have created the two management partitions we have been discussing (vCenter and HW Management). There are also some Guests templates I have created. One important thing to notice from this screenshot is that the first blade can access both shared SAS LUNs (for the records it can also access its own dedicated/local Storage1 VMFS volume):

 

The next picture confirms that both blades can access the shared LUNs created. This allows all VMware advanced features such as VMotion, DRS, HA etc:

 

Here we will attempt a VMotion of the HW Management partition running on esx1 onto the other host in the cluster:

 

The Guest is being moved from one host onto the other. Notice the status bar at the bottom:

 

And here the Guest has moved and it's now running on esx2 as you can see from the Summary pane (and the status bar at the bottom):

I truly believe that the BladeCenter S is a piece of technology that is sometimes under valuated. There is an enormous potential in it that many people haven't fully exploited. It's really what I would describe as a no-compromise Enterprise "pocket" data center. Not so much "pocket" after all because if you think that an HS21XM blade could support, on average, some 15/20 VMs (depending on the workload), we are talking about a 7U Enterprise solution that could support around 100 VMs. Far more than what an average SMB shop might require.

Massimo.

I have been working in IT for about 17 years now, 14 of which at IBM. Since the first day I was immediately exposed to the concept of a centralized IT where everything is fully controlled, fully secured, fully automated and easy to manage within the data center boundaries; on the other hand whatever sits outside of the server room should be dumb and wouldn't require any (major) maintenance tax onto the IT organization. For those that have been around for a while this exactly describes how a mainframe operates (more or less).

"Unfortunately" (you can speculate on the apexes if you want) I have built my career at IBM on something that sits exactly on the other side of the spectrum compared to the mainframe: that is the x86-based server business (was PC Servers, was Netfinity, was xSeries, is now System x / BladeCenter). That's why I have enjoyed, in the last few years, looking at the mainframes as the holy grail (or the polar star) where I'd like to push my "little" x86 servers.

So why is the distributed IT broken? Simply because I think businesses have sold their soul to the evil as they compromised things like control, security, automation and low costs of operations for the nirvana of flexibility and low acquisition costs that came with x86 servers (and PCs). And being this model a client-server model it has affected both the x86-based server portion of the data center as well as the (even more distributed) client environment. Client-Server here doesn't strictly pertain to the architecture of the applications but it rather pertains to the devices one will end up managing no matter what the application architecture is: the application of choice might be Web-based but at the end of the day most likely the IT organization will be running the web server on an x86 Windows or Linux box and the end-user browser will be accessed on a fully featured PC/Laptop running a Windows client OS. It's going to be a Client/Server world anyway no matter the application architecture.

In this brief post I just want to show a couple of proof points of this broken IT model. The first one is a screenshot of a "server" I found during a local customer visit. Ready? Fasten your seat-belt please:

Now, this is not a guess, this is for sure (I did ask) a Microsoft Software Update Services (SUS) "Server". While the first sticker (on the green bazel) says "Test..." the other one features a "NON SPEGNERE" that  means "DO NOT POWER OFF" so those of you that are thinking this was a sort of quick and dirty trial on the desk... should be thinking twice about it. A couple of additional things you might want to notice are that this "server" was physically located on an office desk so it means that the x86-based portion of that data center basically left the actual physical data center rooms and has had ramifications outside of it (very scaring). The second thing to notice is that by no means this is a small SMB shop (I have seen production MAIL servers at those accounts that were even worse than this); no this is a big enterprise customer with many thousands of (actual) servers. Definitely if such big organizations are doing things like these, what's going on in "our" server rooms (and outside of them!) is pretty scaring to say the least.

So much for the server side of the things. How about the clients (desktops/laptops)? Do you remember those zero-maintenance 3270/5250 terminals we all used to access our AS/400 and mainframe programs? Well I took this other picture a few days ago and while it's not as scaring as the other above it tells a lot about where we have got with desktop/laptop management:

It literally says:

--------------------------------------------------------------------------------------------------------------------------

Distribution point for 1GB additional memory (RAM) to install Lotus Notes 8.0.1

The laptop needs to be Powered Off! Not Hibernated!!!

--------------------------------------------------------------------------------------------------------------------------

The scaring thing about this is that the organization going through this massive process has roughly 9.000 employees. If you compare this (little example) to the way a central processing unit with dumb terminals used to work you start getting the feeling about how much broken things are in the x86 (client-server) space.

Now I am 100% sure we won't go back to those days (nor I am suggesting that we try to do that) also because no one would want to give up with the GUI experience for a green character interface (how the h%&l can I watch YouTube on a 3270 terminal?) but yet clearly something needs to be done. The good news is that there are technologies that will allow IT organizations to do this and get to the point where they do not need to trade-off control, security and other important data center aspects to get the flexibility and experience end-users demand (and expect) in the 21st century.

Imagine... a world where your SUS "Server" will just be a service running in your server room (or someone else's server room out in the cloud) that doesn't require a "dedicated server" in your data center (and not even a dedicated desktop in the office - can you believe it?) and where your e-mail client update won't pre-req anyone to go to the office (and waste half a day) to get an additional 1GB of memory....

You may say I am a dreamer, but I am not the only one (where did I hear this?).

Massimo.

So finally it happened. Hypervisors are (essentially) free. I remember the very first engagement I had with VMware technologies some 8 years ago; that was the ESX 1.1 (beta) time frame: we did a Proof of Concept and closed the deal with a very satisfied customer... While they were very happy about the achievements they have always taken the opportunity to remind me how expensive VMware (i.e. ESX 1.1) was. Well, time goes by I guess and what used to be a large chunk of the project expenditure it is now a piece of (business) commodity. "Business commodity" meaning that hypervisor vendors are no longer going to make money out of it, which is different than being a "technology commodity". Well I guess I will save the "control point concept" for another post: it's a long discussion - interesting though.

Back on track.

I am currently doing some research on virtualization vendors positioning in the x86 space and on July 24th Mike DiPetrillo posted a very interesting thought about the implication of making ESXi (yeah Mike, it's ESXi, no longer ESX 3.5i) free of charge. I suggest you read it carefully, along with all the interesting comments, on line here. It's a long thread but if you are among the 99.99% of x86 customers in the SMB space wondering "should I use VMware or Microsoft technologies to virtualize my datacenter?" I strongly suggest that you go through it. Mike did a great job (well other than getting the official brand name of his flagship product wrong... ;-) sorry Mike, I had to say that) in setting the stage.

I am not going to repeat what's in the post (as I have assumed you read it at this point) but this is what he came out with (in blue pen) in order to virtualize some 30 Windows servers on as few as 3 physical hosts.

Need:
Microsoft -- VMware

Basic Consolidation:
$3,000 -- FREE

Centralized Management:
$3,500 -- $2,995

Basic Advanced Features
(Backup and Patching):
$7,260 -- $2,995

There have been a number of comments regarding the fact that Mike used additional fee-based products (for <patching> for example) whereas many customers would be fine with free tools such as WSUS (Windows Software Update Services). Many might also argue that you can't buy VMware products without software subscription and support (which I think Mike didn't take into account) whereas you can buy MS products without those. I am not interested in this micro-level details since, at the end of the day, getting to an apple-to-apple comparison is going to be impossible given the fact that both software vendors have offerings that can hardly intersect with each other: it would be like to try to find the face of a sphere... the sphere has many faces... actually an unlimited number.

I am probably one of the most agnostic persons around when it comes to the virtualization software to be used as I don't have any vested interested in any of the parts involved. As long as customers and end-users can get "the most for the cheapest" I am the happiest person on this earth that's why I welcome so much VMware and Microsoft engaging at this level to provide more value at a more reasonable cost.

Having this said there is something in this analysis that I want to challenge (and not necessarily to put either one vendor or the other in a bad light - I want to do that based on my experience and for the sake of customers). Specifically I want to challenge the assumption that High-Availability should be taken out of the picture. I have always advocated that one of the primary reasons for which customers virtualize is for easy HA and DR. I have written about this feeling many times; here and here for example. So I wanted to re-run Mike's number taking into account Windows Server 2008 Enterprise Edition (which includes MS Cluster Server) and VMware VI3 Standard Edition (which includes VMware HA). I am not a master in pricing but I understand from Mike's post that Win 2008 EE is $4,000 so that's the number I am going to re-work with. On the VMware front things start getting a bit cumbersome. The best option (I think - other suggestions?) would be to use the "Standard" Accelerator Kit which is the counterpart of the "Foundation" Mike used. The "Standard" Accelerator Kit comes at $5,995 but it includes VMware Virtual Center Foundation and "only" 4-sockets VI3 Standard licenses (the "Foundation" Accelerator Kit has 6-sockets licenses). So I have to add another $2,995 for the additional 2-sockets VI3 Standard license a-la-carte (yes apparently, by chance, the 2-sockets "VI3 Standard" license comes at the same price of the "VI3 Foundation" Accelerator Kit).

So the new table, which includes High Availability functionalities, would look like this:

Need:
Microsoft -- VMware

Basic Consolidation:
$12,000 -- FREE                                                                -> Win 2008EE x 3                                                                  --   ESX 3i

Centralized Management (includes high availability):    -> Win 2008EE x 3 + Systems Center VMM Workgroup       --    VI3 Std Accelerator Kit + VI3 Std 2-sockets 
$12,500 -- $8,990

Basic Advanced Features (includes high availability)    -> Win 2008EE x 3 + Systems Center Suite Enterprise         --    VI3 Std Accelerator Kit + VI3 Std 2-sockets 
(Backup and Patching):
$16,260 -- $8,990

So even if you add the mandatory subscription and support costs to the VMware column they continue to lead (substantially?) in terms of price / features. I want to underline again that someone might argue that some of the fee-based MS features are not strictly needed as the same result can be achieved paying less compared to what's in the table above. I'll leave it to you to work out the micro-details and perhaps you might find out that the MS stack might make more sense to you and your specific situation.

I have also to say however that many people have mentioned that Mike didn't take into account the soon to be released stand-alone Hyper-V version for $28. While it's true that this version can change the dynamics of the first table above, it is my understanding that this specific version will not support Enterprise features such as MS Cluster Server so it cannot be used to alter the pricing dynamics of the second table.

However, what I struggle to fully get is related to the last comment Mike did on the post:

>NOTE: Windows licenses were not calculated into the costs since we assumed that the average SMB customer will continue to use and run their existing Windows Server 2003 installations which they already own the licenses for. >Based on lots of conversations with analysts, press, bloggers, and customers this is a safe guess for the next 1 - 2 years as Windows Server 2008 gets adopted. If you were to calculate in licenses costs then the best license to use >would be Windows Server 2008 Datacenter which allows for unlimited VMs no matter which solution you use. You should then subtract $12,000 $3,000 from the Microsoft column in each example and add $6,000 to BOTH >columns (Microsoft and VMware) to get the cost for early adopters of Windows Server 2008.
 

And then again in the comments section:

>Good point about OEM licenses. I tried to keep most of the complexities of Microsoft licensing out of this post, but yes, if you have an OEM license then you cannot reassign it to another server unless you bought Software >Assurance for that license within 90-days of original purchase. I will make a note of this in the blog. I will also put on the to-do list to provide some insight into the complexities of Microsoft licensing.

>The OEM licenses could impact the cost model in 2 ways:

>1) If you decided you were going to stick with Windows 2003 for some time to come then you would purchase new Windows 2003 licenses for your VMs. You would still end up purchasing Windows 2008 licenses as well for the >Hyper-V hosts and not for the VMware hosts. The only exception would be if you purchased Software Assurance with the Windows 2008 licenses in which case you get downgrade rights and could run Windows 2003 in your >VMs. All of this gets complex since Software Assurance is only available through certain Microsoft license agreements which are not always present with SMB customers (our target example in this post).

>2) If you decide to go ahead and upgrade to Windows 2008 then the note at the end of the blog post still holds true on the impact to both columns.

>Thanks again for pointing this out. I for one really do hate OEM licenses since they're so restrictive. Cheaper - yes, but restrictive.

 

Mike is right to assume that most customers would continue to use Windows Server 2003 for some time to come. He is also right that, assuming a complete re-licensing needs to be done, Windows Server Datacenter is the best option given its "unlimited VMs policy". Last but not least he is also right in mentioning that usually these "upgrades" paths for the OEM versions are not available for the SMB customer set.

I think he is missing some specific scenarios and he has some wrong assumptions though . When you license a host with Windows 2008 Datacenter not only you have unlimited virtual machine licenses, but you also have license entitlement for the underline virtualization technology (i.e. Hyper-V / Parent Partition). This means that, if for some reasons (see below) the customer needs to re-license all the 30 Windows instances he is virtualizing, he is going to license everything with Windows 2008 Datacenter which includes both virtual machines and Hyper-V entitlements for the host. Notice that the customer could then install Windows 2003 guests as you have downgrade rights. This is an intriguing scenario because basically you are buying one (Datacenter) license from MS that entitles you to use both the virtualization layer as well as the guests. For the VMware scenario in parallel the Datacenter license is still the one that makes more sense but the problem is that VMware wants his piece of the pie now (in addition to the piece customers have to buy from MS). This makes essentially the MS hypervisor solution REALLY FREE in the comparison of the two technologies.

So assuming to take into account the Guests licensing, in specific situations where you have to re-license them, we are looking at:

Need:
Microsoft -- VMware

Basic Consolidation (includes Guests licensing through Windows 2008 Datacenter - 6 sockets x $3,000 each):
$18,000 -- 18,000                                                               

Centralized Management (includes high availability):
$18,500 -- $26,990

Basic Advanced Features (includes high availability)
(Backup and Patching):
$22,260 -- $26,990

You can also normalize this chart removing the $18,000 price for the Windows Datacenter licenses and the table would end up in listing only the "virtual infrastructure" costs:

Need:
Microsoft -- VMware

Basic Consolidation (assumes but does not include Guests licensing through Windows 2008 Datacenter - 6 sockets x $3,000 each):
FREE -- FREE                                                               

Centralized Management (includes high availability):
$500 -- $8,990

Basic Advanced Features (includes high availability)
(Backup and Patching):
$4,260 -- $8,990

Again, based on my (limited) licensing know-how, these two new tables hold true for:

1. Customers that cannot re-use OEM licenses

2. Customers deploying a new virtual infrastructure, only have older Windows versions (i.e. Windows 200) and want to use newer versions in the guests (i.e. Windows 2003 and Windows 2008)

3. Customers deploying a new infrastructure from scratch

So basically the two statements above made by Mike do not always hold true:

> You would still end up purchasing Windows 2008 licenses as well for the Hyper-V hosts and not for the VMware hosts

> If you decide to go ahead and upgrade to Windows 2008 then the note at the end of the blog post still holds true on the impact to both columns

Assuming my understanding is correct (but I might be missing something) under these assumptions the MS solution stack comes in at a much cheaper price (especially if you account for mandatory VMware subscriptions and support fees). So the first question that needs to be answered is: does this analysis make any sense?

If it does, then the second real question that needs to be answered is... how many customers fall in the 3 scenarios described above that favor the MS licensing model?

How many faces does a sphere have?

Massimo.

Lately, there have been many discussions on the Internet and on various forums regarding the implementation of HA clustering technologies (namely and primarily Microsoft Cluster Server) within virtual machine environments (namely and primarily VMware infrastructures). Many customers are still treating virtual machines as if they were standard Windows servers (or Linux for what that matters) so this does make sense.

However there is a trend in this industry that is shifting typical infrastructure services from the multi-purpose operating systems into the virtual infrastructure. The top of the iceberg of this trend is called Virtual Appliances. While many view Virtual Appliances as a starting point of something big and new I really see them as the natural result (big and new) of this trend that is... turning the hypervisor into a so called Data Center OS. I have discussed this trend in a presentation that I did at VMworld 2007 in San Francisco and that you can access here.

If you stop for a minute and think about what it is happening in this x86 virtualization industry, you'll notice that many infrastructure services that were typically loaded within the standard Windows OS are now being provided at the virtual infrastructure layer. An easy example would be network interface fault tolerance: nowadays in virtual environments you typically configure a virtual switch at the hypervisor level, comprised of a bond of two or more Ethernet adapters and you associate virtual machines to the switch with a single virtual network connection. What you have done in this case is that you have basically delegated the virtual infrastructure of dealing with Ethernet connectivity problems. This is a very basic example and there are many others like this such as storage configuration/redundancy/connectivity.

These two pictures should graphically outline this trend:

 

(for higher quality pictures please refer to the presentation linked above)

Back on track, one of these infrastructure services that is about to migrate from within the multi-purpose OS where the application runs all the way down into the virtual infrastructure is the High Availability service. In the VMware vocabulary this is called VMware HA and this is a piece of code/intelligence that is part of the VI3 offering and whose purpose is to protect virtual machines from host failures. Basically what happens in this case is that, should a host fail, all virtual machines running on top of that failed host get automatically restarted on surviving nodes being part of the same VMware HA Cluster. However many readers would point out that there are at least a couple of very important architectural differences in how VMware HA compares to Microsoft Cluster Server implemented within virtual machines:

• In the case of VMware HA there is a single instance of the virtual machine (with the application) to be protected. The VM is being started on a given node of the cluster given the status of the others (availability and resource utilization). Many people still think that the software stack loaded in the virtual machine is a Single Point Of Failure (imagine a Service Pack upgrade that goes wrong for example and you will have an unplanned downtime of the VM and in turn of the application). On the other hand a "virtual" MSCS solution requires two independent Windows nodes (virtual nodes in this case) so that should any problem occur within the software stack of a node it won't affect the availability of the application that can be restarted on the other virtual node.

• In the case of VMware HA, you are really only monitoring the status of the physical server. Should a physical server go down the virtual machine is restarted on another node of the cluster. This scenario doesn't cover the software stack status within the VM nor, obviously, the application status within the VM (it must be noticed that VI3.5 introduced experimental support for monitoring the status of the OS within the VM via VMware Tools heartbeat check-points). On the other hand in a Microsoft Cluster Server solution you would typically be able to be protected by physical host failures (obviously) and you also would be able to monitor the application status so that a given service can be restarted onto another MSCS node should it fail to start on the "primary" node even if the node has not failed.

This picture should outline the differences of these two approaches:

 

I guess you can easily depict the philosophical differences between the two approaches. The first one is more traditional and tends to treat virtual machines as we have been treating physical servers in the last 10 years, applying the same practices and technologies. In the second picture, the philosophy is more innovative and tends to treat a VM as a simple object which leverages the new virtual infrastructure capabilities.

We are clearly at an inflection point now where many customers that used to do standard cluster deployments on physical servers (which was the only option to provide high availability) are now arguing how to do that. They now have the choice to either continue to do so in virtual servers as opposed to physical servers (thus applying the same rules, practices and with little disruption as far their IT organization policies are concerned) or turning to a brand new strategy to provide the same (or similar) high availability scenarios (at the cost of heavily changing the established rules and standards). The reason I am saying we are at an inflection point is because I really believe that the second scenario is the future of x86 application deployments, but obviously as we stand today there are things that you cannot technically do or achieve with it. Plus, there is a cultural problem from moving from an established scenario to the other.

The following table tries to summarize advantages / disadvantages of both approaches:

Characteristics                                                                                          HA Cluster within the VM                                 HA Cluster at the virtual infrastructure level

Easy Deployment                                                                                      Not True / Can't be achieved                                                          True / Can be achieved

SW stack redundancy                                                                                    True / Can be achieved                                                        Not True / Can't be achieved

Application Monitoring                                                                                  True / Can be achieved                                                        Not True / Can't be achieved

"Guest OS independent" high availability                                              Not True / Can't be achieved                                                           True / Can be achieved

Allows to apply traditional practices and IT standards                                True / Can be achieved                                                        Not True / Can't be achieved

Allows to decouple appl functionalities from HA functionalities             Not True / Can't be achieved                                                          True / Can be achieved

Easy to implement / inherit DR properties                                               Not True / Can't be achieved                                                          True / Can be achieved

 

These are a few of the characteristics many users are currently debating. Again you can depict from the above that delegating this infrastructure service (i.e. HA) to the virtual infrastructure is a better way to implement a data center...at least in my opinion. Assuming proper and effective backup/restore procedures can be implemented for your virtual environments, assuming that you don't need strict application monitoring (or that HA clusters at the virtual infrastructure will improve over time) and assuming an IT organization can adapt easily to new deployment methods and standards... it's obviously where you want to go in the long term.

It is interesting to notice that there are a number of limitations in deploying an MSCS solution in a VMware VI3 environment: one is the fact that the VMDK files corresponding to the C:\ drive of the virtual machine nodes need to reside on a local, non-shared VMFS volume of a given ESX host - which is typically a small partition on the local hard drives that also contains the hypervisor code. On top of this there are a number of other limitations but it suffices to say how bad and not very flexible a Microsoft Cluster Server solution implemented on top of VMware VI3 can turn out to be, with no VMotion of the virtual nodes themselves given the non shared disks.

Another problem associated with the usage of HA software packages within virtual machines is that VMware tends to randomly pull in and pull out support for this at every minor and/or major infrastructure release update. Sometimes I am wondering whether these limitations imposed by VMware are due to technical challenges or to strategic politics from VMware to undermine the minds of those customers that want to keep their traditional practices. In fact this underlines the nature of the VMware strategies which is clearly not only that of introducing a hypervisor between your physical box and your legacy software stacks, practices and standards... their strategy is to literally scramble the entire data center in terms of software stacks, practices and standards. And it's not necessarily a bad thing if you think that these stack, practices and standards are not optimal (as I do).

At this point I also must remind readers that MSCS is an example that might be confusing for the simple fact that this is the very same technology that MS will be using to cope with this new trend. The idea is that, instead of using MSCS within a couple of virtual machines as we described above, they will be using MSCS to act as the High Availability mechanism for Hyper-V similarly to what VMware HA does for ESX. These pictures should clarify the idea:

 

Last but not least I also should mention that VI3 and MSCS are respectively examples of an implementation of an HA solution at the virtual infrastructure level (VI3) and an example of an HA software package at the virtual machines level (MSCS) that I have been using throughout this document to describe the concept. There are other technologies that can be mapped to the same concept and the list hereafter is an attempt to mention some of these options:

Virtual Infrastructure solutions w/ HA capabilities                                           High Availability Software Packages for setup in Virtual Machines

                 VMware Virtual Infrastructure                                                                                           Microsoft Cluster Server (MSCS)

      Microsoft Virtualization (Hyper-V w/ MSCS)                                                                                        Veritas Cluster Server                                              

                VirtualIron Extended Enterprise                                                                                                            ..................

       Citrix XenServer (HA module in roadmap)

                             ..................       

This oversimplifies a very complex matter; for example one could notice that VCS (Veritas Cluster Server) could be used either within a virtual machine environment (as reported in the table above) or as an alternative to VMware HA at the virtual infrastructure layer - similar to how MSCS can be used either within virtual machines or in conjunction with the Hyper-V parent partition. Interestingly enough, in such a context (i.e. used at the virtual infrastructure layer), VCS is potentially able to monitor application status provided the proper Veritas agents are loaded within the virtual machine guests...although this challenges the benefits of a deployment like this being potentially Guest OS agnostic.

Obviously all this discussion strictly pertains to typical HA scenarios where you have an application that deals with and manipulates data, and for which you need a shared storage solution. In all situations where the application is stateless and high-availability can be achieved load balancing multiple instances of it (a good example is a farm of web servers), then both high-availability and scalability is inherited by the layout of the application deployment and doesn't require any "infrastructure HA assist" (be it at the virtual infrastructure level or within the virtual machine).

In the end my suggestion is that users try to evaluate the pros and cons of the "legacy" option vs. the pros and cons of the "new trend", which leverages virtual infrastructure capabilities so that they can take educated decisions. Emotionally I do like the second option much more because it's.... better. But I perfectly understand many IT organizations have their own problems jumping on the wagon right away. By the way, I am totally for virtualization, but realistically I wouldn't rule out the potential situations of keeping some particularly critical x86 workloads on a physical MSCS cluster if that is required. Some organizations also like the idea of implementing N+1 clusters where you can protect N independent physical servers using a single virtualized host on top of which run N virtual images which are the MSCS nodes counterparts of the physical systems to be protected. While this sounds like an interesting scenario - and it is for some situations - it involves the same supportability and limitation concerns we have discussed above.

 As a matter of fact, closing this long post, I have realized that I am ok with everything .... but with using HA software packages within virtual machines running on top of virtual infrastructures.... It's just too complicated, too risky, too cumbersome... too "no way."

Massimo.