[INFO] vCloud Director Fast Provisioned Catalog Virtual Machines

A while back I was messing around with Fast Provisioning in vCloud Director and I noticed something I wanted to dig a little deeper into. My Co-Worker Cormac Hogan (@VMwarestorage) also wrote a little about this as well which does a great job showing the linked clone aspect. Also William Lam (@lamw) wrote up some nice scripts to find the linked chains. However, it took me up until now to get my home lab back into a clean state to test things a little differently specifically with how these interact with the vCloud Director Catalogs. The premise of what I am looking at is a very simple setup, but could change some operational ideas about how and when you enable Fast Provisioning, which is a great, and handy thing to have in test and development environment. However, you need to understand a little about how they work before you check the box to enable them.

There is a couple of things you need to know first about vCloud Director Fast Provisioning.

  1. It is Enabled on a PER organization vDC level so it is either on or off.
  2. ONLY System Administrators can consolidate Fast Provisioned virtual machines, Organization Administrators cannot
  3. Once disabled existing machines will remain fast provisioned

The real key that I wanted to look at here was deploying items back and forth from the catalog with the feature enabled. So what I setup was pretty basic.

  • Master Organization with a published catalog and Fast Provisioning DISABLED
  • Customer organization with local catalogs and Fast Provisioning ENABLED
  • Both Organization vDC’s are Pay-As-You-Go for reference
  • The template is CentOS 6.2 minimal exported/imported from vCenter.

The rest of this post will be various operations in certain orders to see what happens with and without fast provisioning enabled on a consumer organization.

The Shadow Virtual Machine

In some cases, in order for fast provisioning to work it will use the concept of a Shadow VM on each datastore where a linked clone will live, but in many cases it may not get created for some time. This VMware KB has a lot of good information, and a couple of key points taken from it are as follows:

  • The source template virtual machines are called primary virtual machines
  • Shadow virtual machines are created on demand
  • Subsequent copies to the same datastore are fast
  • Org Admin/User only sees the ‘source’ virtual machine. Shadow virtual machines are an implementation detail that are only visible to vCloud Director administrators.
  • Shadow virtual machines stored in System vDC
  • Shadow virtual machines disk space billed to the service provider

Shadow virtual Machines are only created once a clone needs to be placed on a storage volume different from where the original one is located. Until that needs to happen everything else is done on the same storage.

Initial vApp Template Import Into the Catalogs

The first thing I wanted to see was taking an OVF, and importing to each catalog to see what happens. Obviously on the Master organization the catalog item will be imported as a thick copy, but I was not 100% sure on the Fast Provisioned vDC. Interestingly, From what I saw both vApp templates were brought in as full copies into the catalog by the initial import from OVF.

However I actually tested COPYING the vApp template from the Master Catalog to the Org’s local catalog. In this case the copy was actually a linked clone. It created the initial snapshot and then made the local Org’s catalog version a linked clone to the original just as if it was deployed to the cloud itself. I found this interesting as it leads me to something we will discuss later about updating and re-adding items to the catalog.

The Use Case

What we see is a pretty common use case for why I am testing this. It has also been asked how does someone deal with patching and updates once these catalog’s are linked together. The consumer wants to deploy the Guest Operating System from the provider’s published catalog, customize it, and save a copy to their local organization catalog. The consumer’s organization is enabled for Fast Provisioning, but they may want to ensure their local catalog chain does not link back to the master catalog.

Deploying from vCloud Director Shared and Local Catalogs

Where this gets really interesting is on the deployment of each version of the vApp Catalog item. Now we are going to only work off the Master Shared Catalog since that’s what most people would do. The first time a deploy operation of a vApp Template to a Fast Provisioned vDC is requested, the original virtual machine is put into snapshot mode. This means the actual virtual machine that is the catalog vApp in the master organization is now in snapshot mode indefinitely. If there happens to be a Shadow VM required, that would be created, then a snapshot taken. At this point forward anything deployed from that catalog virtual machine will be deployed from the base snapshot.

Now that we have a deployed vApp as a linked clone, we can update it, patch it, add new applications to it, whatever we want to treat it like a normal virtual machine. Let’s say we want to then save this to our LOCAL catalog at this point. When we make that copy to the local catalog, we will get a full copy however there is a catch. The full copy appears to be a full copy of the deployed virtual machine’s Delta Disk as you can see in the info below. We can also see this from the ParentHint information that the machine in the local catalog is also linked to the consumer’s deployed virtual machine itself.


Let’s take this one step further, and now deploy a vApp from this new local catalog virtual machine. What we see here is that the newly deployed virtual machine from the local catalog is in turn linked to the virtual machine in the catalog. This means that as this consumer edits, saves, and re-deploys they are always creating links from the previous virtual machine.

Below we can see the disk headers for each virtual machine in the various steps. One thing that I want to point out is you will see the “parentFileNameHint” which shows that each disk is linked to a disk in the previous step.

Master Catalog VM (Orange)

# cat CentOSMinimal\ \(ff68332f-8a07-4739-9c75-4d945dbec22b\).vmdk
# Disk DescriptorFile
# Extent description
RW 33554432 VMFS "CentOSMinimal (ff68332f-8a07-4739-9c75-4d945dbec22b)-flat.vmdk"

First Consumer VM (Green):

# cat CentOSMinimal\ \(14775f16-7089-4a8e-8993-3045bed173e7\).vmdk
# Disk DescriptorFile
parentFileNameHint="/vmfs/volumes/4f316743-5041e641-2229-005056890004/CentOSMinimal (ff68332f-8a07-4739-9c75-4d945dbec22b)/CentOSMinimal (ff68332f-8a07-4739-9c75-4d945dbec22b).vmdk"
# Extent description
RW 33554432 VMFSSPARSE "CentOSMinimal (14775f16-7089-4a8e-8993-3045bed173e7)-delta.vmdk"

Consumer Local Catalog VM (Blue):

# cat CentOSMinimal\ \(f308e901-60b2-406b-88a1-fb3bcc58540a\).vmdk
# Disk DescriptorFile
parentFileNameHint="/vmfs/volumes/4f316743-5041e641-2229-005056890004/CentOSMinimal (14775f16-7089-4a8e-8993-3045bed173e7)/CentOSMinimal (14775f16-7089-4a8e-8993-3045bed173e7).vmdk"
# Extent description
RW 33554432 VMFSSPARSE "CentOSMinimal (f308e901-60b2-406b-88a1-fb3bcc58540a)-delta.vmdk"

Consumer Re-Deployed Catalog VM (Purple):

# cat CentOSMinimal\ \(f5b612e7-512b-4730-aff8-ee9489012ffe\).vmdk
# Disk DescriptorFile
parentFileNameHint="/vmfs/volumes/4f316743-5041e641-2229-005056890004/CentOSMinimal (f308e901-60b2-406b-88a1-fb3bcc58540a)/CentOSMinimal (f308e901-60b2-406b-88a1-fb3bcc58540a).vmdk"
# Extent description
RW 33554432 VMFSSPARSE "CentOSMinimal (f5b612e7-512b-4730-aff8-ee9489012ffe)-delta.vmdk"

So what happens then if the GREEN virtual machine is deleted from the vCloud Director user’s cloud?  That’s a great question, since you would hope that the underlying disk would not be removed.  That is in fact the case.  The underlying disk remains on storage consuming space since there are other machines, namely the one in the catalog that is linked to it.

Something to Consider – Break The Chain!

Based on now knowing how some of this works, you want to decide how you to handle fast provisioned Org vDC’s since once they are enabled you can see how everything from that point on will be based on links from the previous virtual machine. Something many providers and consumer organizations alike may want to do is break the link chain of their local catalog so they are always deploying from a full copy within their local organization catalog.

This can in fact be done with a few steps. Remember in the beginning when I stated that only a System Administrator can consolidate? That comes into play here, as once a vApp is checked into the local catalog a system admin OR something with system administrator credentials, like vCenter Orchestrator or other scripting method, can consolidate the virtual machines in the consumer’s local catalogs. Below you can see the vCloud Director interface option on the virtual machine in the consumer’s catalog. Using a service to do this for you would certainly be a better way to go.

Consolidating the customized vApp (Green) first however, will convert the virtual machines to full copies before you save it to the local Catalog, but the copy in the catalog would still be linked. You want to Consolidate in the catalog after for best results.  The advantage to this is that the consumer’s can all deploy in this fashion and in theory the provider can have very little linked to their original templates allowing them to patch them and replace them as they need:

  • Start with the provided templates
  • Customize them
  • Add them to their local catalog
  • Consolidate in the catalog to make full copy
  • Remove the original linked customized vApp

To accomplish this of course you need some custom intervention and have your consumer’s deploying via custom tools that interface with things like vCenter Orchestrator. Either way it can be done such that the original virtual machines have no links to them and the consumer’s all have the links to their own local copies. Figuring out all the hooks to leverage the API’s to make this happen…..well….that’s for you to figure out.

Remember that vCloud Director is smart enough to know that if you delete the item from the catalog or the user’s cloud if there are virtual machines linked to it, the base disks will remain on storage. They will not be removed from storage until all linked VMDK’s are eventually removed. So you can deploy, patch, and remove a catalog virtual machine from vCloud Director, and the linked ones will still function.

About Chris Colotti

Chris is active on the VMUG and event speaking circuit and is available for many events if you want to reach out and ask. Previously to this he spent close to a decade working for VMware as a Principal Architect. Previous to his nine plus years at VMware, Chris was a System Administrator that evolved his career into a data center architect. Chris spends a lot of time mentoring co-workers and friends on the benefits of personal growth and professional development. Chris is also amongst the first VMware Certified Design Experts (VCDX#37), and author of multiple white papers. In his spare time he helps his wife Julie run her promotional products as the accountant, book keeper, and IT Support. Chris also believes in both a healthy body and healthy mind, and has become heavily involved with fitness as a Diamond Team Beachbody Coach using P90X and other Beachbody Programs. Although Technology is his day job, Chris is passionate about fitness after losing 60 pounds himself in the last few years.


  1. Have inherited a vCD implementation with Fast Provisioning enabled, two large vAPPs, each with over 20 VMs so over 40 VMs on the same 2TB datastore.  The SAN is Hatachi VSP and VAAI is working.  When the datastore started showing over provisioning above 200% and utilization above 98% we decided to use storage migration to move some  VMs to other LUNs, knowing it would fully expand the disk for the VMs.  We may want to just disable Fast Provisioning at this point and distribute the VMs around to the datastores.  Any issues in disabling  Fast Provisioning after the fact?

    • No you can disable it on the Org vDC after the fact it just means anything still linked will remain linked until you move/Copy them or consolidate them and no NEW vApps will be deployed with Fast Provisioning.

      FYI if you used the vCloud API to do a storage vMotion it would not inflate them it would have created a shadow VM and re-linked them. If you do it via the vSphere UI then you get the full clone on the SVmotion.

      • Thanks for the quick update.  Had to do a cold migration from the vSphere UI but all looks good.  To clean up the remnants of the linked clones will I now need to shutdown the vAPPs and do a move to a new vAPP?  Would it be worth the effort to break the vAPP into smaller vAPPs and move into those?

  2. Chris,
    As always I enjoy reading your article. It’s not often I read something and realize I have been doing it correctly. This article was more of a confirmation for me that I am learning this stuff correctly. There is one thing I wanted to share and it may not be as important. When we update/patch our catalog items we deploy them to the admin org (the org that hosts the catalog) and then patch them. Operationally it would be a good idea to always consolidate before copying to a catalog. As a side not to that I thinks its important to realize that if you have 20+ items in your catalog then every update you perform will create shadow VM’s or leave the original disk out there. So initially that is an overhead to the SP. It is a small overhead in terms of using Fast Provisioning I would like to see something like Composer for vCloud to manage these images. I would be interested in seeding all the different ways folks are managing their clones. Speaking of which, VCO is becoming an invaluable tool.

    • One thing to note. When William and I looked at Consolidating BEFORE you go into catalog, the issue is that the catalog item will STILL be linked to the now consolidated version in the user’s cloud. The only way to really make sure the catalog version is NOT linked is to consolidate after you copy to catalog when Fast Provisioning is enabled on the org vDC.

Leave a Reply

Your email address will not be published. Required fields are marked *