I kept saying for years that vCD should have had more visual support and network layout diagrams in the UI to make it easier to understand and digest that richness. When I sit down with partners and customers to discuss the technology I don’t show the vCD UI.

I rather prefer to use a whiteboard and draw diagrams that often look like, logically, the old good vSphere maps. Do you remember them? How nice.

As part of new responsabilities I am taking inside VMware, I am trying to get a bit deeper on the API side of this cloud thing.

I thought it could have been a good exercise to try to implement a sort of “vCD maps” tool. For the records I end up calling it LiquidDC, more on this later.

A few weeks ago I sat down with my partner in crime Andrea Siviero to build something for real. This was mostly a learning exercise for me on how to code a web application leveraging the vCD APIs. The majority of the coding was done by Andrea. Credit goes where credit is due.

The technical background

A few weeks ago VMware released a fling called Silverlining. That fling contains a few things. In particular it contains a (limited) JavaScript SDK for vCloud Director and a brand new consumer UI for vCloud Director.

So we leveraged the SDK along with some other open source libraries such as JQuerymobile, JQuery and VivaGraph. The figure below illustrates the packaging of Silverlining and how we leveraged it to build the LiquidDC utility package.

We essentially took the overall structure of Silverlining (in particular the JavaScript SDK), complemented it with additional libraries, got rid entirely of the Silverlining UI portal and built from scratch our own new UI.

The end result is a brand new HTML5/JavaScript application.

What LiquidDC does

LiquidDC allows you to connect to a vCloud Director 5.1 tenant and, as an output, it will generate a graphical layout of the network subsystem (and more). The utility allows the user to enable and disable the visualization of certain relationships. We have implemented the following relationships:

• VMs to vApps

• VMs to Organization Networks

• Networks to Edge Gateways

LiquidDC will also visualize the relationship of Organization Networks and Edge Gateways with External Networks.

Let’s take, for example, my IT20 vCD organization hosted at Stratogen. If I look at it from the vCD UI, I can depict my organization has one Edge Gateway called Routed Network. Note the name may be misleading as it’s not really a “network” strictly speaking, but rather a gateway where routed networks connect to.

Note this Edge has 6 L2 networks connected to it. You can check how many of them are outbound connections to External Networks by looking at the Properties of the Edge.

You can check how many of them are networks available inside the virtual data center by clicking on the Org VDC Networks tab:

To add confusion, one of the Organization Networks is called Routed Network, just like the Edge Gateway. In a particular scenario like this it is very difficult to not get confused looking at the UI.

I can conclude that my Edge Gateway (again, called Routed Network) has 5 Routed Organization Networks connected to it. The 6th Edge vNIC (shown above) connects to an External Network (in this case it represents the Internet) which is the interface that connects the Edge Gateway to the outside world.

We are not done yet. There is also an additional network inside my vDC that isn’t connected to anything. It’s the Isolated Network. VMs connected to this network can only talk to each others, but not go anywhere else.

Last but not least, as if the confusion was not enough, there is also a Direct Connect Network available in my vDC that represents direct access to the External Network (Internet). Essentially Stratogen entitled the IT20 organization to connect VMs directly on the Internet segment without having to go through the Edge Gateway. Note that if two organization do this they will end up with VMs on the same L2 segment.

I have to say this is very far from being intuitive for someone that isn’t experienced with vCD . And it isn’t very intuitive for me either, to be very honest. Not to mention the troubles when you need to describe this (for training, demo or PoC purposes) to someone that isn’t very much into the parlance vCD uses. This is when a whiteboard becomes very handy.

Enter LiquidDC!

Below is a screenshot of how the same complex rich networking plumbing described above renders in LiquidDC. Note that the VMs to vApps relationship is set to off by default to simplify the first view.

It is now a lot easier to describe to a vCloud Director novice user what he/she can do with the platform., isn’t it?

The tool is also capable of showing, in a similar graphical layout, the relationships between catalogs and vApp templates in those catalogs. In the picture below you can see an organization private catalog with one template and a cloud public catalog with a fairly big set of templates.

Note that when you click on an object a list of details appears on the right hand side. This is, at the moment, a raw list of attributes (associated to the object) that we get from the REST APIs. We haven’t spent too much time to properly parse, select and format those details. They are pretty raw. The vApp template doesn’t have a lot of these details but if you click on other objects the details are a lot richer than this. See the demo below.

Another cool thing is the Search Object field where a user can search dynamically for a string match against the details mentioned above. In the picture above, for example, I have searched in the catalog layout view for “wordpress” and LiquidDC is dynamically highlighting (with a red circle) the vApp template that contains that particular string.

The details pane and the search capability are available in the network layout view as well. Imagine, for example, being able to search for all networks that have a default gateway that matches “192.168″. Very powerful.

Hybrid comes true

We often hear hybrid cloud being defined as the possibility to move workloads seamlessly from private to public and viceversa.

That’s a key characteristic of a hybrid cloud implementation but it’s not the only angle to look at the matter.

To me, hybrid cloud also means the ability to use the same tools and know-how to manage platforms and infrastructures regardless of where they are hosted (on-premises or off-premises).

And by that I don’t mean having to implement a monster overlay software that may cost 2M$ and 2 years to get deployed. By that I rather mean being able to manage raw dispersed infrastructures, public or private, using and reusing the very same single native API call, the very same native script, the very same native command line.

That’s the interesting part of LiquidDC. You can connect to the real production Stratogen cloud as demonstrate above, or you can also choose any other of the 200+ vCloud Powered or vCloud Datacenter partners based on characteristics like for example:

• Geographic location

• Service level

• Network configuration requirements

• Catalog content

• Particular add-on services

• Pricing

In addition to that you can obviously connect LiquidDC to your local private cloud. I have for example used the tool to visualize the network layout of my IT20 organization hosted at my local private cloud (a lab in the office). As you can see in the picture below the end-point is 172.16.100.205.

Not enough?

I have also used LiquidDC to connect to the vCloud Evaluation Service. Note I don’t have control over the name of my organization and one (2215) was automatically generated for me when I enrolled last year .

In order to find the vCloud API end-point of the evaluation service, you have to login into the custom portal and, from there, open the standard vCD UI. There you can see what the URL is. I also had to create (self-service) a new organization administrator account to be able to connect with the tool (the default admin user won’t let me connect directly, based on the quick test I did).

Enough? No, not enough.

Even more interesting, I was able to connect LiquidDC to one of the zones of the newly announced VMware vCloud Hybrid Service (currently in limited beta). This is not the same thing as the vCloud Evaluation Service mentioned above. Note I had to obfuscate the end-point of this service as it’s not publicly available at the moment.

I think this is pretty cool and, if nothing it’s been an interesting exercise.

The funny thing is that it wouldn’t take too much (all relative) to improve LiquidDC to show more than one single organization in one single cloud in the same UI. Perhaps with VPN relationships as well?

Something like this.

Isn’t this the single “pain” of glass everyone would love to have? And it’s only roughly roughly 400 lines of JavaScript code (without comments)! It’s not a Frankencloud by any means!

LiquidDC use cases

So what would you use LiquidDC for? As I said and Andrea have developed the tool as a coding exercise. However I see a few practical use cases for it. Some are listed below.

- LiquidDC may be a great training and demo tool to illustrate the complexity richness of the vCloud Networking subsystem. Instead of getting on a whiteboard and draw all possible networking configurations nuances in front of someone that doesn’t know vCD one could create the plumbing of an environment including External Networks, Edge Gateways, Organization Networks (Direct Connected, Routed and Isolated) and eventually connect dumb VMs to those networks. LiquidDC can then visualize real-time the layout of that network topology which is far easier to “get” compared to the out of the box vCD UI experience.

- LiquidDC may facilitate basic operations for small customers with small vDCs hosted in public clouds or private clouds. Navigating through the vCD UI may require dozens of clicks to get to the object you need to manipulate or get a particular information from. LiquidDC has what I refer to as a great “time-to-object” (at least compared to the native vCD UI). The search capability is very powerful and can help a lot in this respect.

- LiquidDC could serve as a basis for private cloud administrators and public SP that would like to provide this add-on service to their tenants. If I stretch my imagination a bit I can see SPs taking this code, making it better and more stable and hosting it in their facilities hard coding their end-points. This would allow them to give their tenants an alternative view to browse their organizations and this could be a differentiated service for them.

LiquidDC deployment scenarios

The fun didn’t end with writing the code.

As I said this is a traditional HTML/JavaScript application. For good or bad.

In order to make this whole exercise even more interesting, we decided to distribute it in a couple of ways. A hosted version and an on-premises version.

Did you know you can upload an HTML/JavaScript application to CloudFoundry and host it there? I didn’t think this was possible but Andy Piper, one of my fellow colleagues at VMware, documented a way to do just that.

So LiquidDC is, right now, up and running on CloudFoundry at liquiddc.cloudfoundry.com! Make sure you read below the instructions on how to use it (RTFM!).

I and Andrea are also going to make it available on GitHub hopefully soon. I just need to clean up the code a bit and remove all the embarrassing comments in it. In reality I’d like to document as much as possible the source code so that you know what we were doing and hopefully make it easier for you to modify it if you want to. I’ll update this post when the code is available for download.

Finally, we did not spend time to package this tool so that it could be installed on the vCD cells. Silverlining does come with such a setup utility though. You may try to install Silverlining on vCD and manually change the files (essentially replacing the Silverlining portal with the LiquidDC code). This is really just a after thought I had while writing this blog post. It would need to be vetted.

Instructions and Limitations

Being this a JavaScript application all the cross-domain calls limitations apply. Since this is somehow a derivate of Silverlining ,which has the same limitations, you can use the tricks that William Lam already documented.

At a minimum you’ll need to open your browser with security disabled.

Optionally, if the cloud you are connecting to is using self-signed certificates, you need to accept self-signed certificates in a browser window (very likely situations for demo and PoC environments).

A few known gotchas to take into account.

• We have noticed weird behaviors when you have vApps and VMs that have failed to deploy in the organization you are trying to connect to

• It’s always a good practice to reload the application in the browser whenever you try to re-connect (either to the same organization or to a different organization)

• VMs that are not connected to any Organization Network will render in the graphic as if connected to a dumb non existent network called “none”

• VMs that are connected to a private vApp Network will render in the graphic as if connected to a dumb non existent network called “none”

• VMs that have more than one vNIC will render with only one vNIC

• I have primarily used and tested LiquidDC with Chrome for Mac with the proper flag to disable web security. I haven’t tested other browsers / client platforms.

These hold true for LiquidDC version 0.9.8.5 (the latest available at the time of this writing).

The controls and exception management in the application is… non existent. All in all this tool has gone through very limited testing. And it’s been tested against a very limited number of vCD use cases so we are certainly not considering a lot of exceptions.

I have created a short 4 minutes video that will allow you to see how it works end-to-end, just in case you have problems connecting to your cloud but yet you are curious to see it in action.

If nothing, at least you’ll appreciate why we called it “liquid“!

Massimo.

Update (April 12th): the open source code has been posted on GitHub and can be downloaded here.

This industry has achieved a high level of maturity in terms of best practices (and tooling) for backing up and restoring workloads running on vSphere virtual infrastructures. As we introduced an additional layer on top of vSphere (vCD) we broke, so to speak, some of the tools and many of the best practices. Even more challenging, we introduced concepts that didn’t exist before in a virtualization scenario (cloud providers and cloud consumers).

People tend to always give a crisp yes / no when faced with the question “can you backup/restore workloads running in vCloud Director”? I think the matter is more complex than that. It really boils down to what you want to do (more on this later).

I was tasked (I actually volunteered) to double click on this. Admittedly I started this effort with a short minded view that was (on the line of) “let’s find out which backup and restore tools integrate with vCloud Director”. As I started to lay out the content it became very clear that I was trying to find out the micro-details without having clear the potential macro-architectures and big picture. I started to lay out the context and I thought that making it public would help gathering more feedbacks and getting valuable inputs on how to proceed. What you will see next is (more or less) part of the content I am working on. It goes without saying that this are the informal rants of a single cloud architect. This is not a VMware paper (as is) and you shouldn’t refer it as such when pointing to this blog post.

Introduction to the vCloud Director Storage Layout

The figure below shows a high level view of the vCloud Director storage architecture.

There are a lot of considerations missing in the picture above in terms of how the storage stack is constructed in vCloud Director 5.1 (for example Storage Profiles, Provider vDCs, vSphere clusters, etc.) but there is enough information to describe the backup and restore process (and associated challenges).

First of all one can depict the multi-tenancy nature of vCloud Director where a single datastore/LUN (and host, for that matter) can be securely shared among different tenants (aka organizations).

vCloud Director presents a certain amount of (abstracted) storage to the tenant as a property of the organization vDC (aka Org vDC) the tenant has been assigned to. The tenant can consume that storage by creating VM disks as a property of a VM. The tenant does not care where that abstracted pool of storage resources are coming from.

Another important thing to notice in this simplified diagram is the fact that different actors can access the same resources at different levels. For example:

• A tenant can access and can manipulate resources in its organization vDC whereas a cloud administrator can manipulate all resources across all tenants

• A tenant can access a file on the VM file system by means of a Guest OS operation whereas a cloud administrator can access the same file mounting the VMDK at the ESXi host level

• A tenant can perform limited manipulation on VMDK files via the vCloud APIs (e.g. independent disks, new in vCD 5.1) whereas the cloud administrator can fully manipulate them using traditional vSphere mechanisms

Infrastructure Visibility

This parameter, later used to characterize backup and recovery solutions, describes the level of access a given individual may have in a vCloud Director stack.

vCD uses a role-based model to assign proper rights to users. In the context of this document we will divide the cloud world in two macro roles: providers and consumers.

In vCD language, they are the cloud administrator and the organization administrator.

Note: We will consider roles like vApp user and vApp author being a subset of the organization administrator role and, as such, with a slightly limited visibility compared to the latter. We will just consider the organization administrator as the cloud consumer.

We introduce here two key concepts in cloud operations. These may be relevant in general for cloud but they are indeed very relevant for vCD cloud deployments.

These concepts are above-water visibility and below-water visibility. The water line alluded here is the line that separates cloud tenants from cloud administrators.

It is important for cloud administrator and cloud consumers to pay attention to this parameter (visibility) because that determines whether a given backup solution they are (respectively) building or consuming is available out of the box without customizations and on any vCloud Director deployment available.

“Above-water” Visibility

With above-water visibility (or consumer space) we refer to all of those operations that can be performed by a vCD tenant (specifically by an organization administrator) with an out of the box vCD. The emphasis here is on vanilla and out of the box.

These are all standard operations that any vCD tenant can perform regardless of the vCloud Director implementation (private or public that is).

This is a list of operations that, for example, an organization administrator can do above-water:

• Creating a “backup server” inside the tenant to backup locally the files (inside the OS) of the production VMs

• Manually copying vApps either in the same PvDC or in different PvDCs

• Programmatically copying vApps either in the same PvDC or in different PvDCs

• Leveraging independent disks to attach / detach VMDK files to stateless VMs

• Leveraging independent disks (through attach / detach) to create Guest OS mirrors of production VMs.

Many of these approaches are usually typical of “design for fail” cloud models and don’t usually fly very well with customers with an Enterprise mind set.

Also, a missing out of the box object storage service in vCD limits the above-water backup and recovery use cases. An alternative workaround would be to setup a proxy inside the tenant that can backup to a third party public object storage service.

For example an object storage can be configured as a target in some traditional backup and restore tools or some third party public object storage services provide appliances (aka storage gateways) that can act as a proxy between a private set of servers and the public object storage service.

All of the above is considered above-water since this is something the tenant can implement without any interaction with the cloud provider and, more importantly, without any particular vCloud Director customization or extension.

This applies to any vCloud Director based cloud instance.

“Below-water” Visibility

Describing below water visibility (or provider space) is fairly easy because it is, essentially, full visibility into the cloud stack. This is only available to the cloud administrator and, assuming the vCloud Director administrator is also the administrator of the infrastructure underpinning it (which is often the case), this includes visibility into a variety of tools and layers including, obviously, vCenter Servers.

The cloud administrator is the owner of the entire stack and can perform any operation at any level in the stack. This is obviously true within the boundaries of what it is supported by the integration of the various products in the vCloud Suite.

There are for example tasks that, while the cloud administrator can perform at a lower level, are not supported as they may break the layers above. Some of these tasks, for example, include (source: vCAT 3.0.2):

• Editing virtual machine properties

• Renaming virtual machine

• Disabling DRS

• Deleting or renaming resource pools

• Changing networking properties

• Renaming datastores

• Changing or renaming folders.

In the context of backup and recovery of consumer workloads, operating at this level of the stack requires careful planning by the cloud administrator.

This is a list of operations that, for example, a cloud administrator can theoretically do below-water:

• Backing up / restoring files inside tenants via VMware VADP

• Backing up / restoring VMDKs inside tenants via VMware VADP

• Backing up / restoring VMs inside tenants via VMware VADP

• Backing up / restoring vCloud vApps inside tenants via VMware VADP

• Other objects manipulation aimed at saving the state of those objects using vCenter administration level of access.

Some of the operations above, particularly the restore of vCloud objects, require particular attention and best practices.

 Most vCloud implementations will vary below-water. This is true for many other operations but it is certainly true for backup and recovery operations. While there is a set of basic core functionalities a cloud admin can perform using VMware tools at this layer, most implementations will be complemented by peculiar backup and restore software products and, perhaps, particular configurations of the same backup and restore software products.

So while we consider the above-water zone to be consistent and standard across all vCloud Director deployments, we anticipate the below-water zone to be specific and peculiar for every deployment.

Backup and Restore levels

This is the second parameter that we will use later to characterize and segment backup and recovery solutions.

This is straightforward and describes the “what” in the backup and restore equation. What objects do tenants need to backup (and be able to restore)?

These objects and levels are discussed below in this section. The following picture summarizes them graphically.

File Level

This is the most atomic thing in the cloud consumer space that the tenant may want and can backup (and restore). It can’t get more granular than that. There isn’t a lot to say about it. A file inside a Guest OS file system is just a file.

Disk Level

This refers to the VMDK file associated to a given VM. It’s fair to see the VMDK as the drive of the VM. Note that by backing up the VMDK you are essentially backing up the entire state on disk of that Guest OS. In Microsoft Windows parlance, it’s like backing up the entire c:\ drive.
The relationship between the VMDK and the files discussed above is 1:many.

VM level

This object includes the VMDK content as well the metadata describing the virtual machine. A VM is really the collection of the content of the (virtual) disk as well as surrounding data that describes the characteristic of the VM (number of vCPUs, amount of memory, number of vNICs, etc.). This information is saved in the vmx file (which sits next to the VMDK file, in the same folder).
The relationship between the VM and the VMDK can be 1:many (limits apply, albeit it is often 1:1).

vApp level

This object describes the service (or the workload). A vApp is usually referred to as a collection of VMs but there are more to it than that. A vApp includes information such as vApp Networks (and associated network and security levels), VMs start and stop order, etc.
vCD vApp metadata and vCD VMs metadata are also part of the properties of the vApps.
The relationship between the vApp and the VM can be 1:many (limits apply)

Managed Service Vs. Self Service

This is the last parameter that we will use to characterize a backup and restore solution for vCloud Director consumer workloads.

At first this may sound like a duplicate of the above-water and below-water segmentation but it is not.

The infrastructure visibility parameter speaks more to the implementation of the cloud environment and the out of the box capabilities.

This segmentation speaks more to the operational aspect of performing backup and recovery of consumer workloads.

While it would be easy to mapping the above-water concept with self-service and mapping the below-water concept to managed services the reality may be more complex.

For example a given cloud service provider may offer managed services using above-water capabilities.

Or, even more interesting, a cloud consumer could experience a self-service experience using below-water capabilities (by means of third party portals or API extensions that the cloud administrator can expose to the tenant and that are not available out of the box with a vanilla vCloud Director setup).

Cloud Provider Managed Service

This is the scenario where the cloud administrator owns the operational aspects of backing up (regularly) and restoring (on a need basis) consumer workloads on behalf of the cloud consumer.

This is true regardless of:

• Whether the cloud administrator uses an above-water (less likely) or a below-water (more-likely) strategy

• What level of backup and restore is required (file, disk, VM or vApp)

In this scenario the cloud administrator usually have a set of policies in place to backup the consumer workloads (depending on the agreed SLAs) and the cloud administrator personnel perform the restore. Depending on the contract in place this could happen without consumer interaction or the consumer, by opening a ticket with the cloud service provider, could trigger the restore.
In this scenario the self-service aspect of cloud is not leveraged and exploited.

Cloud Consumer Self Service

In this scenario the tenant is fully in control of the backup and restore operations.

This is true regardless of:

• Whether the cloud consumer uses an above-water or a below-water strategy

• What level of backup and restore is required (file, disk, VM or vApp)

There is typically no interaction between the cloud administrator and the tenant and every backup and restore operational aspect is available to the cloud consumer.

Note the nature of backup operations may vary depending on the implementation details.

For example in an above-water backup and restore strategy the tenants are responsible for building and consuming their own solution.

However, when a tenant is consuming, in self-service, a below-water solution implemented by the cloud service provider, backup operations may be driven by:

• Pre-defined policies (e.g. all vApps placed in a given virtual datacenter will have a pre-defined backup policy)

• Self-service policies (e.g. the tenant can interactively assign vApps to particular policies interacting with the cloud via third party service portals or API extensions)

Backup and Restore: Solutions Characterization

Why is this important? Ideally every backup and restore solution we will discuss in the context of this document can be characterized by this triplet we have defined:

• Where? (above-water or below-water)

• What? (files, disks, VMs or vApps)

• Who? (tenant self-service or provider managed services)

The triplet above isn’t useful to describe the inner technical details of any backup and restore product. However it is very useful to describe the outer characteristics of any backup and restore solution.

Ideally, before talking about the actual implementation, cloud architects should be able to characterize a solution by the where / what / who parameters.

This is true for architects building clouds (e.g. “our vCloud Director based backup and restore strategy will allow tenants to restore VMs and vApps by opening a ticket with us. We will then leverage some of our below-water features not exposed to the tenants”).

Similarly, architects consuming clouds should be able to query potential cloud service providers about their backup and restore services using this framework (e.g. “we are looking for a vCloud Director based service that would allow us to restore files, disks and VMs in self-service leveraging below-water features”).

Note that, for the most part, the infrastructure visibility aspect (below-water, above-water) isn’t usually something a consumer may want to call out as a “requirement”. Ideally the consumer would always want something to be “above-water” because that means the solution could be implemented on any vCD based cloud should they choose another cloud provider. However, the reason for which a tenant may specifically ask for a below-water functionality is because they have enough know-how of the vCloud stack to require a particular and more efficient solution than what a tenant may be able to achieve above-water.

In summary, we have been introducing the concept of above-water and below-water.

We have then introduced the list of objects that could be a target for backup and restore operations.

Last but not least we have introduced the notion of self-service and managed services.

The following picture represents a self-service solution.

The following picture represents a managed services solution.

That’s all (I can disclose). This is the framework I have been working on lately. As often happens to me, I can’t tackle a very simple problem without having to put it into the bigger picture to contextualize it. Sorry about that.

While I do understand that many people are interested in “does backup product xyz talk to the vCloud APIs”, I fear a simple yes or no doesn’t cut it and doesn’t put those people in a position to build a proper backup and restore solution for their vCloud Director based cloud.

Now, the next challenge is how to lay out (in a meaningful way) the research and unstructured work I have been doing to double click on actual solutions. What I have in mind right now (subject to change) is to describe in greater details a certain number of solutions and architectures (4? 6? 10?) that could be considered most common and best practices and characterize each of them with the where / what / who framework I discussed above.

This would let VMware customers and partners come up with their own additional solutions / combinations that they could characterize with the same framework. Just a thought at the moment.

Any comment or feedback that you may have, I am all ears.

Massimo.

I guess we can summarize the bulk of those interviews in the following quote from one of the articles: “Specifically, Casado says we can expect a hypervisor-agnostic network virtualization platform that could be marketed as an independent product.”

This obviously brings up the tedious topic of… can a platform vendor really become platform agnostic? More on this later.

This goes back to a Nicira slide I built a few months, before VMware bought Nicira. This is the slide I am referring to:

Note I was using that slide for a slightly different argument (which, in turns, was going back to my ABC of Lock-In theory). However what this picture was (implicitly) conveying is that, in order to be in a particular spot of the infrastructure, you do need to be agnostic to the stuff that surrounds you.

vSphere, in the context of a server hypervisor, is agnostic to the hardware and to the Guest OS it supports. Similarly, the Nicira NVP needs to be agnostic to the hardware and hypervisors it supports. Or, to steal Martin’s specific way to put it: “…networking is the one thing that you can’t be a unilateralist with. The network touches everything. It’s the network.”

By the way, in the slide above, you can picture vCloud Director, OpenStack (or whatever) instead of vSphere, Xen, KVM (or whatever). Same concept.

Now, let’s go back to the original question: can a platform vendor (e.g. VMware) really become platform (e.g. vSphere and vCloud) agnostic?

Well, after having introduced the theory of The ABC of Lock-In and The Cloud Magic Rectangle I am introducing today, in this post, the theory of the T (or the T theory).

The theory is really very simple and it goes like: it depends where the money flow.

This is how I picture in my head a vendor with a (traditional) platform play and a (new) cross platform play:

The T theory continues by saying: if the vendor is able to make the money in the cross platform play, then the same vendor is willing to concede third party platforms more love (for lack of a better IT word). On the other hand, if the vendor is not able to monetize on the cross platform play, then the same vendor is NOT willing to concede third party platforms more love (and will try to drive and funnel their customers towards the platform play where they can still make money).

Let’s try to make three examples of the T theory.

Did the cross platform play work for IBM? I would say so. IBM is making a lot more profits on the Tivoli product line than it is making on AIX (the mainframe is a tricky story) so they got well past this dilemma of potentially compromising their own platform play by having a cross platform play.

Will the cross platform play work for VMware? Who knows. What we know is that VMware said that there are (a lot of) money to be made in that space though. This doesn’t mean that VMware will push to compromise the platform business with this strategy. However this does tell that, if VMware is able to make money on the cross platform play, potentially compromising the platform play (by working with third parties platform plays) will be worth it. You can go a step ahead and make a parallel between Nicira NVP and DynamicOps but let’s not complicate the discussion too much as there are different nuances there.

Will the cross platform play work for Microsoft? Who knows. What we know is that Microsoft is giving away that piece, arguable a core technology of the data center of the future,  for free. Admittedly I don’t have an MBA but to me this means either one of two things: they will try to move people to the platform (where they are still making money) or they are going to charge for that piece (if they want to be in the true platform agnostic business).

At least this is what the T theory says. I realize, however, there are gray areas in it. I am not going to call out all possible nuances to avoid boring you more than I have done already.

Massimo.

Kidding aside, it will sound complex, but this is the “tax” you need to pay for being able to provide (as a cloud provider) and consume (as a cloud consumer) virtual data centers instead of virtual machines. After all, flying a Boeing 747 is inherently more difficult than driving a Fiat Panda but it really boils down to where you need to go in the end.

This is a summary of how the three models work with vCloud Director 1.5

This is a summary of how the three models work now with vCloud Director 5.1. The yellow cells represent the changes from the previous version of the stack.

So let’s start with the easy part. No change at all for the Reservation Pool model. Easy. Done.

There is only one small change in the PAYG model. Now the cloud administrator can create an Org vDC that is capped not only by the # of VMs but also by CPU and memory resources limits. This is cool because many customers (and Service Providers) liked the PAYG model but they needed to cap the tenant with something that was more sophisticated than the mere absolute numbers of VMs. vCloud Director 5.1 now delivers that capability to cap the tenant on a resources consumption basis. This is of course an optional and additional parameter that doesn’t change the original PAYG model behavior in vCloud Director 1.5.

As you can see most of the changes (and dramas) come with the Allocation Pool model. This has generated some reactions from our customers (and Service Providers). More on this later.

Change or not to change, that is the problem

The vast number of changes have been introduced to support elasticity for the Allocation Pool model. For those of you new to this concept, VMware defines an Org vDC elastic when it can draw resources from all clusters that comprise a Provider vDC. Alternatively, when the Org vDC can only draw resources from the primary (or only) cluster in the Provider vDC it is defined, guess what, non elastic.

In an elastic Org vDC scenario, every time vCloud Director deploys or deletes a VM, it increases or decreases the size of the Resource Pools (from now on RPs) dynamically. And it can do this across different clusters so that the sum of all of those RPs dedicated to the tenant is reconciled at the vCloud Director level which now owns admission control for all of those VMs.

The good news is that, in vCloud Director 5.1, Org vDCs created with the Allocation Pool model are elastic. The bad news is that this requires some changes in the behavior of this model.

With vCloud Director 1.5, admission control and resource governance for the Allocation Pool model was delegated largely (but yet not completely) to vSphere. The drawback was that vCloud Director had to pre-create upfront a RP that mapped statically the characteristics of the Org vDC the cloud administrator was creating. And the only way to do this was to create a single, fixed size RP in the primary cluster of the Provider vDC. In other words: no elasticity.

What effect does this have on VMs deployed with vCloud Director 1.5 in an Org vDC created with the Allocation Pool model? vCloud Director 1.5 would set a reservation and limit on the VM being deployed based on the % of guaranteed capacity defined when the cloud administrator created the Org vDC. For example,  if I have a 10GB of memory Org vDC with 50% guaranteed capacity and I deploy a 4GB VM, vCloud Director will set a 4GB limit on the VM and a 2GB reservation (50% of 4GB).

However, the way vCloud Director 1.5 managed CPU resources was a total different story. vCloud Director 1.5 didn’t set any reservation / limit value on the vCPU deployed in the Org vDC created with the Allocation Pool model. The cloud consumer could deploy infinite vCPUs (well, ok..) and all of them would fight for the capacity of the fixed size RP backing the Org vDC.

vCloud Director 5.1 moves this on-the-fly resource manipulation at the RP level rather than at the VM level. This allows vCloud Director to treat RPs as dynamic entities (without having to create them upfront with a fixed size) and spread those RPs across many cluster.

Wait a moment, it’s easy to move that memory manipulation from the VMs to the RPs. There is still one piece missing though: how can vCloud Director implement the same RP dynamism with CPU resources? How can vCloud Director 5.1 expand and shrink CPU capacity in the RPs as VMs are deployed and deleted? Enter the vCPU speed parameter.

This new parameter available in the Allocation model wizard in vCloud Director 5.1 allows the system to apply limits and reservations for the CPU subsystem at the RP level. Let’s take this example: a cloud administrator sets a vCPU speed of 2Ghz when creating an Org vDC. The Org vDC has 10GHz worth of CPU capacity with a 50% guarantee. The result of this is that, when the user deploys a VM with one vCPU, the system will increment the limit of the RP (or one of the RPs in an elastic Org vDC scenario) of 2Ghz and will increment the reservation of  the RP of 1Ghz.

That’s how vCloud Director 5.1 achieves CPU elasticity and dynamicity with the Allocation Pool model through the vCPU speed parameter.

This is all good but customers and Service Providers started to provide feedbacks. Essentially it boils down to two things:

• if you set a vCPU value too high you’ll end up deploying a limited number of VMs / vCPUs before the system will reach the CPU resources cap of your Org vDC. In the example above (10Ghz allocated, 50% reserved, vCPU=2Ghz) it would be 5 vCPUs.

• if you set a vCPU value too low you’ll workaround the problem above but you will experience low performance initially for the first VMs you deploy. In fact, at steady state, all the many VMs will compete for the same “big” capacity in the Org vDC, but initially that capacity will be very limited as it gets incremented from 0 as VMs are added.

If, for example, the vCPU speed is set at 200Mhz, the first three VMs will deploy in a RP that has a 600Mhz limit (200Mhz x3) and a 300Mhz reservation (50% of 600Mhz). Even if these three VMs will peak at different times (which is likely) each of their individual vCPUs won’t be able to use all the nominal capacity of the Org vDC (10Ghz). As the VMs being deployed increase, the perceived behavior will get closer and closer to what it was with vCloud Director 1.5 (that is all vCPUs fighting for a big bucket of resources).

Ironically the “problem” mentioned in the first bullet above existed already for the memory subsystem with vCloud Director 1.5. In other words the cloud consumer couldn’t oversubscribe memory. As cloud consumers added VMs, those VMs memory reservations and limits would count against the RP reservation and limit backing the Org vDC up to a point where the system would refuse to deploy more VMs. Apparently the consensus seems to be that this is ok for memory but it’s not ok for CPU. For the CPU subsystems it appears that setting a hard and predictable limit on number of vCPUs that can be deployed by a tenant is not acceptable. And all this regardless of the fact that those “infinite” vCPUs were deployed in a bucket that had limited and finite capacity anyway. Not sure how much of technical and how much of psychological there is in this discussion.

For example, I think this CPU enforcement is a good way to set an average “vCPU to core” ratio so that the tenant doesn’t deploy a number of vCPUs that highly exceed the ratio that that the cloud administrator has determined to be the most optimal. Consider the example below that I am stealing from an internal discussion (not my wording but I like the way it is explained):

Assume that a tenant wants to purchase 100 GHz CPU and 100 GB memory guaranteed in the cloud with an option to burst 4X opportunistically.
We need an allocation pool Org VDC of 100 GHz of CPU reservation and 100 GB of memory reservation.
If the hardware backing the Provider VDC has a core frequency of 1 GHz (say), you can set the vCPU to GHz mapping to 1 GHz.
Next up, you will need an estimate of how much CPU over subscription you / customer want to do. Assuming 4:1 over subscription ( = 4, same as 4X burst), you can allocate up to 400 VMs (reservation * over subscription / vCPU) with 1 vCPU from this Org VDC. This requires an allocation of 400 GHz.
So, to configure the allocation pool Org VDC, you would set it up with an allocation of 400 GHz and 25% guarantee so that you get 100 GHz CPU reserved. Setting vCPU = 1 GHz will allow all the 1 vCPU VMs to consume up to 1 GHz (core frequency) and a user can provision up to 400 VMs in this Org VDC.

I think this makes sense. I like it. But the fact is that a few customers started to complain vocally about this new CPU resource management behavior in vCloud 5.1.

Enter vCloud Director 5.1.1. VMware heard this feedback loud and clear and vCloud Director 5.1.1 introduces a slight change that allows a cloud administrator to, possibly, revert the Allocation Pool model experience to a behavior that is similar to that found in vCloud Director 1.5. In particular, there is only one change that vCloud Director 5.1.1 introduces and that is:

• vCloud Director creates the RP(s) in the cluster(s) with the limit set upfront based on the Org vDC allocated size.

In essence, by pre-setting the RP limit to the nominal size of the Org vDC, the cloud administrator can now set a low vCPU speed value (as this will not be used to increment the RP limit at VM deployment time because it’s already provisioned upfront). What this mean is that the very first VM will find immediately the big bucket of CPU resources it is supposed to draw from.

Note, however, that vCloud Director keeps incrementing the reservation of the RP(s) at VM deployment time based on the vCPU speed setting and the guaranteed % specified in the Org vDC creation wizard. This hasn’t change from vCloud Director 5.1. The other thing that hasn’t changed (based on my tests) is that the vCPU speed cannot be set below 0.26Ghz (or 260Mhz) so when I say “a low vCPU speed value”, 0.26Ghz is the lowest it could get.

This means that reservation of CPU cycles of this bucket is still dynamic and directly proportional to the number of VMs deployed and calculated  from the vCPU speed parameter as well as the % of reserved CPU resources (as it’s in vCloud Director 5.1).  This is deemed acceptable because the assumption is that most clusters are memory constrained. Not reserving CPU at the pool level isn’t going to be a big problem (in most circumstances).

It is important to pay attention to the details introduced in 5.1.1. Because a RP with the total allocated capacity is created on all clusters backing a Provider vDC, this could, potentially, lead the cloud administrator to provision more resources to the tenant than what the tenant subscribed to. For example, a 10 GHz Org vDC based on the Allocation Pool model, with vCloud Director 5.1.1 would result in “n” RPs with a 10Ghz limit where “n” is the number of clusters backing the Provider vDC.

If you are using the vCPU speed  parameter as it is intended to be used with the new vCloud Director 5.1 Allocation Pool model (see example above), the above behavior isn’t relevant. In fact all tenants will have a bigger bucket of shared CPU capacity to draw from but will still be limited in the number of VMs they can deploy and, more importantly, will still be provided with the same reserved CPU capacity as it was with vCloud Director 5.1.

In vCloud Director 5.1.1, unwanted overprovisioning of resources may arise when both the below circumstances are true:

• the Provider vDC is backed by multiple clusters to gain elasticity

• the cloud provider set a very low vCPU speed parameter to bypass the limit in number of VMs that can be deployed

Under these circumstances, the tenants can indeed deploy a very high number of VMs (given that the number of VMs that can be deployed will tend to infinite when the vCPU speed tends to zero). This has the side effect that tenants will have access to a large bucket of overprovisioned resources due to the fact that a RP with the allocation limit is set on every cluster that is part of the Provider vDC. The second side effect is that reservation per each tenant is set to a low number (given it is still proportional to the vCPU speed and the number of VMs deployed) thus leaving different tenants with a potential high number of VMs all fighting for shared resources without proper reservations.

VMware recognizes this is not ideal but the assumption is that many customers and SPs that are using vCloud Director 1.5 are using Provider vDCs backed by a single cluster so the change introduced with vCloud Director 5.1.1 will allow them to upgrade transparently to this release and keep a vCloud Director 1.5-like behavior for the Allocation Pool model. In the future you may see additional flexibility in how to leverage these different behaviors.

This is pretty much about it. And trust me, I gave you the simplified story. There are a lot more details I am not getting into in the interest of time.

OK but what does all this mean for me?

As I am sure you are more confused than when you started reading this post… perhaps it makes sense to put a stake in the ground and underline advantages and disadvantages of the three models with vCloud Director 5.1.

The PAYG model is the most simplistic of the three. This model allows the tenant to scale without pre-configured limits. It does also allow cloud consumers to scale without any contractual agreement on resources. Sophisticated capping mechanisms now allows the cloud administrator to limit a tenant based on number of VMs, CPU and memory resources. One thing to notice is that all VMs in a PAYG are standalone entities that have specific limits and guarantees that can’t be shared with the other VMs in the same tenant. So if a VM is not using all the reserved capacity available to it, that capacity cannot be used by other VMs in the tenant that are demanding more resources. The other typical disadvantage of this model is that it’s based on a first-come-first-served basis. Given the cloud consumer didn’t subscribe to allocated or reserved resources, the system may refuse to deploy VMs at any time depending on the status of resource consumption on the Provider vDC.

The Allocation Pool model is interesting because it allows the cloud administrator (but not the cloud consumer) to oversubscribe resources. The level of oversubscription is set by the cloud administrator at the time the Org vDC is created and the cloud consumer cannot alter those values. The most evident advantage of this model is that the cloud consumer has a set of allocated and reserved resources that has been subscribed (typically for a month). The other advantage of this model is that all VMs in the same Org vDC can share CPU and memory resources inside a bucket of resources that is dedicated (yet oversubscribed) to the tenant. The disadvantage of this model is that the cloud consumer can deploy a finite number of VMs before their total resources hits the limit of the Org vDC.

The Reservation Pool model is radically different from the above two. In this model a Resource Pool is completely dedicated and committed to the cloud consumer. This means that all oversubscription mechanisms are delegated to the tenant thus giving to the cloud consumer the flexibility to choose the oversubscription ratio of resources. The disadvantage of this model is that the cloud administrator cannot benefit from oversubscribing resources at Org vDC instantiation, given the allocated resources to the tenants are 100% reserved. This means that the cloud consumer will have to absorb the cost of this premium service from the cloud provider. Note that the Reservation Pool model (with vCloud Director 5.1) is the only one that doesn’t support elasticity thus further limiting the cloud provider flexibility and architectural choices.

Massimo.

Update: On November 6th this post went through a heavy update. The previous version of the post included some misleading and erroneous information on how vCloud Director 5.1.1 works.