Guest OS Customization in vCHS

calendar Jun 19, 2014 · 12 min read  ·
Share on: linkedin copy

As I was taking notes with some vCHS workload deployment experiments I thought I'd turn them in a blog post. This is more for me to find it easily in 2 months when I will have forgotten pretty much everything I found.

If you are interested in the matter of guest OS customization options in vCHS read on, but I can't guarantee the result.

Customizing a guest OS is a science in itself (the downside of being very powerful). If that guest OS is Windows, then it becomes rocket science.

With all the talking about "application focus" I know that the OS is a pretty boring argument in 2014. However it is still something customers need to deal with from a practical perspective.

Also, this is absolutely mandatory to understand as you begin layering additional concepts and services that do leverage many of these mechanisms (e.g. vCloud Connector virtual machines migrations or DRaaS failover and failback mechanisms).

Background

The premise here is that you basically need to configure your guest OS in a couple of situations:

  • when you deploy your VM from a template in a catalog (be it public or private)
  • when you need to reconfigure that VM at runtime after deployment (e.g. for changing its network settings or the guest OS password)

What I am going to describe below isn't necessarily how I would like things to work. It is how things work (disclaimer: at the best of my understanding at least).

The assumption VMware is making for customizing the guest OS is that there are two class of customizations. I will go head and call them light customization and heavy customization. Warning: this naming convention is my own, not VMware's.

The idea is that the former can be invoked transparently at any time as needed whereas the latter should be invoked one-off (typically at deployment time).

Light customization is about reconfiguring the guest OS for:

  • changing network settings
  • changing computer name

Heavy customization is about reconfiguring the guest OS for:

  • changing password
  • running a customization script
  • changing SID (Windows only)
  • joining a domain (Windows only)

We can debate forever whether changing the computer name should be "heavy" and / or changing a password should be "light" instead. If you have opinions please share them, I am all ears.

Since I love throwing pictures here and there, this is what we have been talking about so far (with color codes: green is light customization, orange is heavy customization):

The (vCD) GUI View of Things

As of the time of this writing, many of the advanced functionalities and life cycle management is done via the vCloud Director user interface (linked from the vCHS portal when advanced configurations are required).

Guest OS customization details are exclusively available (as of today) through the vCD UI. This is what we are going to focus on in this section.

The guest OS customization (whether light or heavy as discussed above) will only occur if the main "Guest Customization" switch is set to on. If the switch is not set to on then nothing happens and the VM is not customized at all. You can set this switch in the properties of he VM:

Often, when you deploy a VM from a catalog for a brand new deployment, this bit is flagged (as in the picture above). The first time you power on a VM (with the Guest Customization flag set to on) everything gets customized.

You can leave the "Guest OS Customization" flag on because, if nothing changes, nothing will happen any more.

If you decide to disable the flag and later you try to change part of the VM configuration that has ramifications into the guest OS configuration (e.g. network settings or computer name) the system will prompt with the following message:

If you set the flag on, the new network settings and computer name will be applied to the guest. If you leave the flag disabled nothing gets applied.

Next is what happens with the guest OS configurations that are part of the "heavy" customization list. Let's say for example that we want to change the guest password and we want to do so from outside of the guest itself. As part of an IaaS automation framework / solution for example (not sure if throwing the DevOps buzzword is appropriate at this point but it will surely make me appear cool if I do). To be noted that, if you later change the password inside the guest, this field will become irrelevant as vCD isn't able to track that change. More on this use case later.

For the records vCHS will offer the following options when it comes to password customization at the infrastructure level:

If you set a different password with the "Guest OS Customization" flag set to on and you power cycle the VM nothing happens. Why is that?

That is expected because password reset is part of the "heavy" customization list and, by default, that customization only gets invoked the first time the VM is powered on in its entire life cycle. Note that while I have only actually tested reconfiguring the password, the same concept applies to changing the domain, changing the SID and running a customization script (none of which I have actually tested in my lab).

There is however a (prescriptive) way to force re-running this "heavy" customization in the vCD UI. You need to open the vApp, go to the VM tab and there you find the option to * Power On and Force Recustomization*:

If you do this, a whole "heavy" recustomization process will occur.

The (vCHS) Portal View of Things

As mentioned above a lot of these details are only visible and actionable in the vCloud Director UI. In the vCHS portal there isn't much that you can see and do other than being able to set the computer name (aka Guest OS Name) at VM deployment time (which will be pushed into the guest OS when the "heavy" customization process is invoked at the very first boot).

The other thing that is being exposed is the guest OS password that you have set:

The APIs View of Things

Last but not least!

All you will see below is based on what I have already described in this blog post and in this other blog post as (one of the many) techniques to navigate through and interact with the virtual data center structure using the APIs.

Upon instantiation via APIs (/action/instantiateVAppTemplate) of a brand new vApp from the vCHS public catalog this is what an API query shows for the VM:

Note: if you set the XML body to deploy the VM but to not power it on, the vApp and the VM will show up in the UI interface as "partially running" and, for some reasons, the needsCustomization bit is set to "false". If you instantiate the vApp with the deploy bit set to "false" the vApp and the VM will correctly show as "powered off" and the needsCustomization bit is correctly set to "true". Careful consideration should be used when instantiating a vApp via APIs as the transition between powered off, deployed and powered on states (or viceversa powered on, undeployed, powered off sates) may have ramifications on the behavior of the guest OS customization routines.

Anyway, as expected, the VM is ready to be customized now. Upon subsequent deployment and power on, everything gets customized. If you run the very same API query against the very same VM now it says:

In the same page, you will also find (in the guestCustomizationSection) whether the guest OS customization flag is on (true) or off (false). This is what the vCD UI reads when opening the wizard in the very first screenshot of this post:

As expected, changing the computer name does change back the guestCustomizationSection bit to "true" and it triggers the "light" customization of the guest OS. I actually already tried this (without knowing it) months ago in these API exercises.

Surprisingly, changing the binding of the vNIC to a different Org Network does trigger indeed the "light" customization of the guest OS but the needsCustomization bit remains set to "false". However, if you take a further look at the XML response when querying the VM via API, you will notice that right above the guestCustomizationSection described above, there is a NetworkConnection section that contains another needsCustomization bit. This bit is in fact set to true upon changing the binding of a vNIC to an Org Network.

I am not sure why there is a VM level needsCustomization and a NetworkConnection sub-level needsCustomization bit that only reports on the network status.

So far we have been playing with configurations that trigger "light" customizations (e.g. network and computer name).

Let's play with a configuration that will change parameters that need a "heavy" customization: changing the password for example. We will assume that all other configurations that require a "heavy" customization work the same way (as I am not testing them).

After changing the password of the VM with the proper API call (e.g. with a REST PUT of /action/reconfigureVm), the VM level needsCustomization bit remains set to "false". Interesting.

It doesn't appear to say anywhere that you need to run a "heavy" customization routine to apply the change you made. As a matter of fact this could become pretty confusing for an end-user: at this point both an API call and a look at the UI (both vCD and vCHS portal) will inform the user of the new password.

However the new password has not yet been configured. I fell into this trap many times.

If you want to fix this problem in the UI you have to force the recustomization of the VM as described at the beginning of this post.

If you want to force this with API interaction, you need to use the customizeAtNextPowerOn API call.

Note: even after forcing the recustomization with the API mentioned above the needsCustomization bit remains set to "false". I haven't so far found a way to determine whether the VM, at subsequent reboot, will be customized with the "heavy" customization routine. In other words there is no way (that I am aware of) to query the VM to know whether it will be customized (or not).

It appears that the VM level needsCustomization bit will only tell if the computer name has changed (since the network change is tracked into the sub-level needsCustomization bit and the "heavy" changes seem to be not tracked at all).

(Apparently) Templates Matter

What we have discussed so far should provide you with enough (yet high level) information on how to build your own solution and VM life cycle strategies leveraging these customization features.

If you are leveraging vCHS global templates, note that some of this life cycle has already been built into these templates and the whole customization behavior is (partially) driven by pre-configurations that VMware has done on these templates.

For example ALL vCHS templates are configured in a way that forces the customer to change the login password at first boot. The way this is achieved is by setting the template to auto-generate a guest OS password and force the change at first login.

With Linux you will have to enter the auto-generated password (reported via the APIs, the vCD portal or the vCHS portal as illustrated above) and then change it immediately after.

With Windows you don't even have to use the auto-generated password because Windows will ask you to directly set a new password.

It is important to understand that, since you are changing the password inside the guest OS, the auto-generated password will still be displayed (via APIs, the vCD portal and the vCHS portal) but it will be useless at this point as that password is now different and vCHS lost track of it at the infrastructure automation level (outside the guest OS).

From what I have also seen in preliminary tests, changing the password at the infrastructure level only really works for Windows guest OSes after the first round of setting a password via manual console input (which doesn't really help automating things - if proven to be true). My assumption is that this is due to how the Windows template has been configured and prepared prior to the upload into the public catalog.

For Linux guest OSes setting a password via vCHS customization features seems to be possible at any time of the VM life cycle. It also goes without saying that boot time of a Windows guest (particularly if it needs to be sysprepped and things like that) is phenomenally higher than a Linux guest.

Note that, at the time of this writing, there is a bug in the vCHS portal that doesn't force the user to change password at first login. This means that if you deploy a Linux VM via the vCHS portal you will not get asked to change the password at first login.

Another thing that I have found interesting from an operational perspective is that some of the catalog templates you will be deploying are not going to trigger the "heavy" customization routines automatically.

Consider this common flow for example:

  • a vApp template gets instantiate from the vCHS public catalog
  • the vApp gets powered on (and the "heavy" customization takes place)
  • the vApp is powered off
  • the vApp is put into a private catalog
  • a new vApp is instantiated from the newly created template in the private catalog
  • the new vApp gets powered on

I would have expected this to result in a run of the "heavy" customization routine because this is, to me, the first time this newly instantiated vApp boots up. However, for somereasons, it doesn't and if you want to re-run the "heavy" customization routines you need to force it on the VM (either via the UI or the APIs). This is something I am stillworking on to fully understand.

Conclusions

I have to admit that this isn't the simplest way to do things. Having this said I have also to say that, before starting to experiment with all this I was a lot more scared andconcerned about how the whole thing worked. There are a few things that I would change here and there (would love your feedback if you have any) but all in all, as it often happensin life, once you know how to do something, those things become fairly easy (and actually quite powerful).

Also, I haven't explored deeply the feature of running a script during guest OS customization. Interesting things could be achieved through this feature as you can read here and here.

This is very much still work in progress for me but I wanted to push out a snapshot of the findings so far. I apologize if it came across like "messed informal personal notes" (which is a perfect definition for what it is).

Massimo.