Liquid Data Center

calendar Mar 18, 2013 · 12 min read  ·
Share on: linkedin copy

Having worked for about 3 years with vCloud Director I have to admit that the networking subsystem is the one that takes more time to digest. Part of this is because it is fairly complex rich. Part of it is because VMware has not done a great job at trying to expose that richness in a simple way to the cloud consumer.

I kept saying for years that vCD should have had more visual support and network layout diagrams in the UI to make it easier to understand and digest that richness. When I sit down with partners and customers to discuss the technology I don't show the vCD UI.

I rather prefer to use a whiteboard and draw diagrams that often look like, logically, the old good vSphere maps. Do you remember them? How nice.

As part of new responsabilities I am taking inside VMware, I am trying to get a bit deeper on the API side of this cloud thing.

I thought it could have been a good exercise to try to implement a sort of "vCD maps" tool. For the records I end up calling it LiquidDC, more on this later.

A few weeks ago I sat down with my partner in crime Andrea Siviero to build something for real. This was mostly a learning exercise for me on how to code a web application leveraging the vCD APIs. The majority of the coding was done by Andrea. Credit goes where credit is due.

The technical background

A few weeks ago VMware released a fling called Silverlining. That fling contains a few things. In particular it contains a (limited) JavaScript SDK for vCloud Director and a brand new consumer UI for vCloud Director.

So we leveraged the SDK along with some other open source libraries such as JQuerymobile, JQuery and VivaGraph. The figure below illustrates the packaging of Silverlining and how we leveraged it to build the LiquidDC utility package.

We essentially took the overall structure of Silverlining (in particular the JavaScript SDK), complemented it with additional libraries, got rid entirely of the Silverlining UI portal and built from scratch our own new UI.

The end result is a brand new HTML5/JavaScript application.

What LiquidDC does

LiquidDC allows you to connect to a vCloud Director 5.1 tenant and, as an output, it will generate a graphical layout of the network subsystem (and more). The utility allows the user to enable and disable the visualization of certain relationships. We have implemented the following relationships:

  • VMs to vApps

  • VMs to Organization Networks

  • Networks to Edge Gateways

LiquidDC will also visualize the relationship of Organization Networks and Edge Gateways with External Networks.

Let's take, for example, my IT20 vCD organization hosted at Stratogen. If I look at it from the vCD UI, I can depict my organization has one Edge Gateway called Routed Network. Note the name may be misleading as it's not really a "network" strictly speaking, but rather a gateway where routed networks connect to.

Note this Edge has 6 L2 networks connected to it. You can check how many of them are outbound connections to External Networks by looking at the Properties of the Edge.

You can check how many of them are networks available inside the virtual data center by clicking on the Org VDC Networks tab:

To add confusion, one of the Organization Networks is called Routed Network, just like the Edge Gateway. In a particular scenario like this it is very difficult to not get confused looking at the UI.

I can conclude that my Edge Gateway (again, called Routed Network) has 5 Routed Organization Networks connected to it. The 6th Edge vNIC (shown above) connects to an External Network (in this case it represents the Internet) which is the interface that connects the Edge Gateway to the outside world.

We are not done yet. There is also an additional network inside my vDC that isn't connected to anything. It's the Isolated Network. VMs connected to this network can only talk to each others, but not go anywhere else.

Last but not least, as if the confusion was not enough, there is also a Direct Connect Network available in my vDC that represents direct access to the External Network (Internet). Essentially Stratogen entitled the IT20 organization to connect VMs directly on the Internet segment without having to go through the Edge Gateway. Note that if two organization do this they will end up with VMs on the same L2 segment.

I have to say this is very far from being intuitive for someone that isn't experienced with vCD . And it isn't very intuitive for me either, to be very honest. Not to mention the troubles when you need to describe this (for training, demo or PoC purposes) to someone that isn't very much into the parlance vCD uses. This is when a whiteboard becomes very handy.

Enter LiquidDC!

Below is a screenshot of how the same complex rich networking plumbing described above renders in LiquidDC. Note that the VMs to vApps relationship is set to off by default to simplify the first view.

It is now a lot easier to describe to a vCloud Director novice user what he/she can do with the platform., isn't it?

The tool is also capable of showing, in a similar graphical layout, the relationships between catalogs and vApp templates in those catalogs. In the picture below you can see an organization private catalog with one template and a cloud public catalog with a fairly big set of templates.

Note that when you click on an object a list of details appears on the right hand side. This is, at the moment, a raw list of attributes (associated to the object) that we get from the REST APIs. We haven't spent too much time to properly parse, select and format those details. They are pretty raw. The vApp template doesn't have a lot of these details but if you click on other objects the details are a lot richer than this. See the demo below.

Another cool thing is the Search Object field where a user can search dynamically for a string match against the details mentioned above. In the picture above, for example, I have searched in the catalog layout view for "wordpress" and LiquidDC is dynamically highlighting (with a red circle) the vApp template that contains that particular string.

The details pane and the search capability are available in the network layout view as well. Imagine, for example, being able to search for all networks that have a default gateway that matches "192.168". Very powerful.

Hybrid comes true

We often hear hybrid cloud being defined as the possibility to move workloads seamlessly from private to public and viceversa.

That's a key characteristic of a hybrid cloud implementation but it's not the only angle to look at the matter.

To me, hybrid cloud also means the ability to use the same tools and know-how to manage platforms and infrastructures regardless of where they are hosted (on-premises or off-premises).

And by that I don't mean having to implement a monster overlay software that may cost 2M$ and 2 years to get deployed<. By that I rather mean being able to manage raw dispersed infrastructures, public or private, using and reusing the very same single native API call, the very same native script, the very same native command line.

That's the interesting part of LiquidDC. You can connect to the real production Stratogen cloud as demonstrate above, or you can also choose any other of the 200+ vCloud Powered or vCloud Datacenter partners based on characteristics like for example:

  • Geographic location

  • Service level

  • Network configuration requirements

  • Catalog content

  • Particular add-on services

  • Pricing

In addition to that you can obviously connect LiquidDC to your local private cloud. I have for example used the tool to visualize the network layout of my IT20 organization hosted at my local private cloud (a lab in the office). As you can see in the picture below the end-point is 172.16.100.205.

Not enough?

I have also used LiquidDC to connect to the vCloud Evaluation Service. Note I don't have control over the name of my organization and one (2215) was automatically generated for me when I enrolled last year.

In order to find the vCloud API end-point of the evaluation service, you have to login into the custom portal and, from there, open the standard vCD UI. There you can see what the URL is. I also had to create (self-service) a new organization administrator account to be able to connect with the tool (the default admin user won't let me connect directly, based on the quick test I did).

Enough? No, not enough.

Even more interesting, I was able to connect LiquidDC to one of the zones of the newly announced VMware vCloud Hybrid Service (currently in limited beta). This is not the same thing as the vCloud Evaluation Service mentioned above. Note I had to obfuscate the end-point of this service as it's not publicly available at the moment.

I think this is pretty cool and, if nothing it's been an interesting exercise.

The funny thing is that it wouldn't take too much (all relative) to improve LiquidDC to show more than one single organization in one single cloud in the same UI. Perhaps with VPN relationships as well?

Something like this.

Isn't this the single "pain" of glass everyone would love to have? And it's only roughly roughly 400 lines of JavaScript code (without comments)! It's not a Frankencloud by any means!

LiquidDC use cases

So what would you use LiquidDC for? As I said and Andrea have developed the tool as a coding exercise. However I see a few practical use cases for it. Some are listed below.

  • LiquidDC may be a great training and demo tool to illustrate the complexity richness of the vCloud Networking subsystem. Instead of getting on a whiteboard and draw all possible networking configurations nuances in front of someone that doesn't know vCD one could create the plumbing of an environment including External Networks, Edge Gateways, Organization Networks (Direct Connected, Routed and Isolated) and eventually connect dumb VMs to those networks. LiquidDC can then visualize real-time the layout of that network topology which is far easier to "get" compared to the out of the box vCD UI experience.

  • LiquidDC may facilitate basic operations for small customers with small vDCs hosted in public clouds or private clouds. Navigating through the vCD UI may require dozens of clicks to get to the object you need to manipulate or get a particular information from. LiquidDC has what I refer to as a great "time-to-object" (at least compared to the native vCD UI). The search capability is very powerful and can help a lot in this respect.

  • LiquidDC could serve as a basis for private cloud administrators and public SP that would like to provide this add-on service to their tenants. If I stretch my imagination a bit I can see SPs taking this code, making it better and more stable and hosting it in their facilities hard coding their end-points. This would allow them to give their tenants an alternative view to browse their organizations and this could be a differentiated service for them.

LiquidDC deployment scenarios

The fun didn't end with writing the code.

As I said this is a traditional HTML/JavaScript application. For good or bad.

In order to make this whole exercise even more interesting, we decided to distribute it in a couple of ways. A hosted version and an on-premises version.

Did you know you can upload an HTML/JavaScript application to CloudFoundry and host it there? I didn't think this was possible but Andy Piper, one of my fellow colleagues at VMware, documented a way to do just that.

So LiquidDC is, right now, up and running on CloudFoundry at liquiddc.cloudfoundry.com! Make sure you read below the instructions on how to use it (RTFM!).

I and Andrea are also going to make it available on GitHub hopefully soon. I just need to clean up the code a bit and remove all the embarrassing comments in it. In reality I'd like to document as much as possible the source code so that you know what we were doing and hopefully make it easier for you to modify it if you want to. I'll update this post when the code is available for download.

Finally, we did not spend time to package this tool so that it could be installed on the vCD cells. Silverlining does come with such a setup utility though. You may try to install Silverlining on vCD and manually change the files (essentially replacing the Silverlining portal with the LiquidDC code). This is really just a after thought I had while writing this blog post. It would need to be vetted.

Instructions and Limitations

Being this a JavaScript application all the cross-domain calls limitations apply. Since this is somehow a derivate of Silverlining ,which has the same limitations, you can use the tricks that William Lam already documented.

At a minimum you'll need to open your browser with security disabled.

Optionally, if the cloud you are connecting to is using self-signed certificates, you need to accept self-signed certificates in a browser window (very likely situations for demo and PoC environments).

A few known gotchas to take into account.

  • We have noticed weird behaviors when you have vApps and VMs that have failed to deploy in the organization you are trying to connect to

  • It's always a good practice to reload the application in the browser whenever you try to re-connect (either to the same organization or to a different organization)

  • VMs that are not connected to any Organization Network will render in the graphic as if connected to a dumb non existent network called "none"

  • VMs that are connected to a private vApp Network will render in the graphic as if connected to a dumb non existent network called "none"

  • VMs that have more than one vNIC will render with only one vNIC

  • I have primarily used and tested LiquidDC with Chrome for Mac with the proper flag to disable web security. I haven't tested other browsers / client platforms.

These hold true for LiquidDC version 0.9.8.5 (the latest available at the time of this writing).

The controls and exception management in the application is... non existent. All in all this tool has gone through very limited testing. And it's been tested against a very limited number of vCD use cases so we are certainly not considering a lot of exceptions.

I have created a short 4 minutes video that will allow you to see how it works end-to-end, just in case you have problems connecting to your cloud but yet you are curious to see it in action.

If nothing, at least you'll appreciate why we called it "liquid"!

Massimo.

Update (April 12th): the open source code has been posted on GitHub and can be downloaded here.