How To: Upgrade vCloud Director in Detail – Part 2

After the VMware vCloud Director, vShield Manager and vCenter Chargeback components have been upgraded, customers can choose to pause the upgrades here, which is recommended. Customers may want to ensure that the vCloud layer is properly running while the backend is still vCenter 4.1 and ESX 4.1. Once they have decided that vCloud Director is still running properly, they can begin the vCenter Server upgrades. It is important to note that the advanced features of vCloud Director 1.5 require a complete vSphere 5 installation, not just the vCenter component.

Assumptions

VMware vCenter Server is integral to vCloud Director; therefore some aspects of the vCloud services will be down during the upgrade. Although the portal may still be up, commands to deploy items to vCenter will not function while vCenter is being upgraded. This means if users have vCloud Director Portal access during the upgrade, they could issue tasks that will fail when the vCenter services are no longer available. It is possible in a vCloud Director scenario where multiple vCenters exist that the portal can remain up. The procedures below assume there is a single vCenter configured in vCloud Director, therefore all users will have no functions. This procedure can be modified as needed to “Disable” only the vCenter being upgraded allowing users to still actively deploy and maintain vApps hosted on remaining vCenters.

Personnel Resources Needed

• VMware vCenter Administrator
• VMware vCloud Director System Administrator
• Database Administrator
• Network/Load Balancer Administrator – To redirect load balancer to temporary page 

Phase II Impact

This phase of the upgrade may cause downtime for the following components.

• vCloud Director Portal and API– The portal should be taken down to prevent vCenter tasks from failing; thus, users will have no API, Portal, or VMRC access
  • This is the case with a deployment where only a single vCenter Server is configured
  • If multiple vCenter Servers are configured then each one can be individually disabled and the portal can remain up and running.
• Custom vCenter Roles- vCenter roles get updated on an upgrade. If you create even copies of the Administrator role for the vCloud service accounts, these will no longer function. You will need to readjust them to have “All Privileges”.
• Nexus 1000v – This may need to be upgraded prior but is undetermined at this time since the new version is not released yet.
• vCenter Heartbeat – This should be upgraded to the latest version that supports vCenter 5.0 and is also backward compatible to vCenter 4.1. 

Phase II Advantages

Performing this phase involves upgrading vCenter, which is required to enable the new features of vSphere 5 and vCloud Director 5.

Upgrading VMware vCenter Server and Update Manager

Because the vCenter Server upgrade is already documented in the vSphere 5 Installation Guides, we will cover only the specific considerations for this upgrade specific to vCloud Director. Refer to the vSphere Installation Guides for detailed upgrade steps. As always backup the vCenter database after stopping the vCenter Services, before proceeding with the upgrade

VMware vCloud Considerations when Upgrading the vCenter Server

There are a few options to deal with vCloud Director during the upgrade of vCenter. Some of the options are better for a setup with multiple vCenters, and others may choose to completely disabled the Cells, or simply disabled the vCenter Servers. Not all of these options need to be used, choose what is right for you.

• Remove user access to the vCloud environment again by redirecting the Load Balancer to a maintenance page if you have only one vCenter or you want to ensure no users access the system
• Use the Cell Management tool (installed with vCloud Director 1.5) to gracefully shut down the services on all cells if you want to ensure no cells are available.
• You can also disable a single vCenter if you have multiple vCenters configured and work on one at a time.
  • NOTE: Just disabling the vCenter in vCloud Director does not prevent the cells from trying to connect to them since they are already configured. For this reason you may still choose to shut down the cells to prevent the reconnection attempts. Disabling only prevents access to the vCenter by tasks.
  • Tool is located in /opt/vmware/vcloud-director/bin

Table 5 – Cell Management Tool 1.5 Options and Arguments, login and help

Managing a Cell

To manage a cell, use the cell subcommand in the format:

[root@cell1 /opt/vmware/vcloud-director/bin]# ./cell-management-tool -u administrator -p Pa55w0rd cell –status
Job count = 3
Is Active = true

As with before you will want to Quiesce first and check the status until tasks are at zero then you can shutdown

NOTE: In some labs these did not seem to return the values on the screen when using SSH, but they did work from a console.

Table 6 – Cell Management Tool Options and Arguments, cell Subcommand

• Stop the Chargeback services; since this also talks to vCloud and vCenter that will both be offline, these should be stopped from the previous steps.
• Back up the vCenter Server and vCenter Update Manager databases
• Begin the vCenter Server and vCenter Update Manager upgrades as you would, using the Installation Guides and Knowledge Base Articles for guidance

Figure 11 – Upgraded vCenter Server Build Number

• Add the vCenter 5.0 Licenses to vCenter Server or these will be in Evaluation Mode
• Restart the vCloud Director services on the cells
• DO NOT RE-DIRECT THE LOAD BALANCER nor allow users back onto the system yet, additional steps need to be taken in VMware vCloud Director.
  

Updating the vCloud Director Configuration

Once vCenter Server has been upgraded and it has been tested with the vSphere Client and the ESX 4.x hosts have updated their vCenter Agents, we can begin the specific vCloud updates to finalize the connectivity to vCenter. Post-install vCloud Director 1.5 is NOT 100% functional with vCenter Server due to the new “Managed Solution” model introduced with vCenter Server 5.0.

If you examine vCloud Director you will see that the vCenter is connected and showing “5.0.0” However, operations for power on, and deploy will actually FAIL with the following errors:

A specified parameter was not correct.

spec.managedBy.extensionKey

Figure 12 – vCloud Director Showing vCenter 5.0.0

The major step that must be taken to fix this error is to “Refresh” the vCenter Server. “Reconnect” will not perform the required functions to add vCloud Director into the new vCenter Server’s Managed Solutions. You can see below from the screen shots that the “Refresh” will add the major functionality to vCenter Server

Figure 13 – vCenter Refresh Command

Figure 14 – Refresh vCenter Warning

Once this step is completed, you will see the properly inserted “Managed By” items in vCenter Server 5.0 and all operations will work. If you look at this screen before the refresh, you will not see any vCloud Director shown. This missing step does not affect vShield Edge functions, as those are not “Managed” by vCloud Director but are managed by vShield Manager

Figure 15 – vCloud Director Showing in Managed Solution

Figure 16 – Virtual Machines Showing Managed By vCloud Director

Once the changes are made you can verify functions are working by deploying a new vApp or powering off a vApp and the commands will work. At this point you can redirect the Load Balancer and DNS back to the VIP for vCloud Director and allow users to access the system. From this point on there should be no other user access disruptions but virtual machines will need to be updated.

Validation Steps to Perform After vCenter is Upgraded

The following two simple steps should be performed to ensure that functions are still working before moving to the next major step. These are basic but will ensure that there is a baseline of functionality as you move forward.

1. Deploy a new vApp
2. Create a NAT routed network

Updating VMware vSphere Update Manager

There is no impact to vCloud Director for this step; however, it is required in order to easily upgrade the ESX hosts to vSphere 5. The upgrade process is documented in the vSphere Upgrade documentation and should be done before moving forward. Clients will also need to download and update the plug-in.

Phase II Completion Verification Checklist