This outlines the steps for performing an “End-to-End” upgrade of a vCloud deployment. “End-to-End” for the purpose of this guide refers to all aspects of the installation from the vCloud Director cells all the way to the deployed virtual machines running inside the vCloud. What this document will not cover is the specific click through steps on certain components, as those are documented in their respective installation guides or release notes. What this will cover are the dependencies, additional steps and considerations needed to maintain the proper working condition of the system at certain stages in the upgrade process and facilitate a smoother process. It is recommended that there are “planned” pause periods between phases to allow for validation and user testing before moving onto the next phase. Although all phases could be completed in a “Big Bang” approach this may add risk and troubleshooting difficulty.

IMPORTANT NOTE: Some of these upgrades may be disruptive to end users, and will require that they do not have system access for a brief period. This is unavoidable, and precautions should be taken for informing users. Running virtual machines will remain available in most cases. This will also NOT address the process for migrating from an Oracle-based vCloud Director Database to MS SQL Server, the process which is currently not supported.

Installation References

The following publications contain information on upgrading each individual component discussed in this document.

• VMware vCloud Director 1.5 Install Guide
  
• VMware vShield Manager Release Notes
  
• VMware vSphere 5 Upgrade Guide
  
  • vCenter Server
    
  • Update Manager
    
  • Orchestrator
    
• Chargeback Release Notes
• Reference all available KB articles as well.

Support and Assistance

Contact VMware Global Support for any assistance and always reference the available Knowledge Base Articles and product installation guides

High Level Process Outline

The upgrade has been divided into four major phases as shown in the order below. After completion of any phase, a customer can “park” indefinitely until they chose to move on.

Table 1 – Upgrade Phases

Each of these phases will be addressed with considerations in separate sections in this document. In testing the upgrade process and considering potential variations in the order of steps, the methodology presented here provided the most stable and phased approach to an upgrade. For example, a customer could elect to only perform Phase I and II and wait on any vCenter or ESXi upgrades and subsequent changes until they wanted the additional functionality. The choice of when to proceed further can be left up to the customer to decide, and they are not left in a state of instability.

There has been a question of upgrading vCenter and ESX first. The downside with upgrading ESX and vCenter before vCloud Director is that the vCloud Director 1.0.1 agent simply will NOT install on an ESXi 5 host. This means if you start the upgrade process with ESX first you will be forced into upgrading vCloud Director planned or unplanned. This may not be the best option for customers that want a truly phased methodical and systematic approach. However, in our tests we observed that upgrading vCenter only to version 5 did not adversely affect vCloud Director in any noticeable way other than Custom Roles are modified and may need to be updated.

Additionally, some customers have mentioned they want to upgrade to vCloud Director specifically for Oracle 11gR2 support or Microsoft SQL Server support, and want to wait on upgrading the vSphere end until they can test it elsewhere. The main point is once you start with vSphere first you will be forced to continue all the way through and have no real phased approach.

Upgrade Checklist

VMware vSphere

Table 2 – VMware vSphere Checklist

Software

Table 3 – Software Checklist

Service Accounts

Be sure to have all the service accounts available with passwords that were used for the original installation. The following table is a reference table of accounts that could have been used during installation and their mapping. Your customer’s use of service accounts may be different, and the key is to understand for each component what account was used, as this can affect the upgrade functionality. In the test lab each component was given their own unique service account in Active Directory (AD) to test the connectivity to other components. Some customers may NOT have used service accounts depending on the installation, but many of the new security features especially in vShield Manager will require them. The table below lists the most common access requirements and the level of access they will need. If these are not in place, they should be created prior to upgrading.

 

Table 4 – Suggested Service Accounts Used

Assumptions

This section also assumes that customer’s will have multiple vCloud Director Cells front-ended by a load balancer in a production-like environment. This also assumes that the transfer folder location was mapped to an NFS share as part of the multi-Cell Requirement. It is also assumed before starting this upgrade that all functions of vCloud Director are working properly. Two key functions would be the ability to deploy a new vApp as well as deploy a NAT Routed network with a vShield Edge.

 

Personnel Resources Needed

• VMware vCloud Director System Administrator
  
• VMware vCenter Administrator – with access to console of vCenter
  
• VMware vShield Manager Administrator
  
• Linux Administrator with Root Access to the vCloud Director Cells and with passwords to cell consoles, cell root login, SCP to cells (to transfer vCD installer and cell management tool),
  
• Windows Administrator with access to the Chargeback server and service account
  
• Database Administrator – For backups, installers, installs, etc
  
• Active Directory Administrator
  
• Network/Load Balancer Administrator – To redirect load balancer to temporary page

Phase I Impact

This phase of the upgrade will cause downtime the following components will be affected by this step in the upgrade process.

• vCenter Chargeback – Version 1.6.2 is only be used with vCloud Director 1.5, although this is not stated in the release notes. Therefore if you upgrade before vCloud Director you will have issues talking to vCloud Director 1.0.1. It is recommended you simply stop the services and proceed to upgrade Chargeback after vCloud Director is fully upgraded.
  
• vShield Manager – vShield Manager usually requires a specific build to work with vCloud Director. Although older vShield Managers will continue to work, you will have to upgrade this as part of this step to enable the new features of vShield.
  
• vCloud Portal and API – The vCloud portal and API will not be available during this phase. It is difficult to determine specific downtime, as it will depend on the number of Cells and size of each customer’s database. This also means the VMRC will not be available to users.
  
• vCloud Director Database – The upgrade does change the schema, so a database backup is important.
  
• vCloud Connector 1.0 – This version will no longer work with vCloud Director 1.5, and possibly not with vSphere 5. The vSphere connection has not been tested, but there is a new completely different architecture for vCloud Connector 1.5 currently in BETA. Just be aware that version 1.0 of vCloud Connector will no longer work.
  
• Rollback – This would require a lot of effort because the database changes are irreversible. It is recommended to back up the database after stopping the vCloud Director Services before moving forward; however, after step 3.5.2 there is no easy path to roll back without major downtime and possible data loss. You should also back up the vShield Manager database using the UI FTP commands. This is the only method of even possibly restoring the vShield Manager for a re-deployment. It is also recommended to take a backup of the vCenter database at this time, as well as during future steps throughout this document as the vCloud Director services get started multiple times throughout the process and can affect the vCenter and vShield Manager databases.

Phase I Advantages

The following is a list of advantages of upgrading to vCloud Director 1.5; however, not all new features will be available upon completing just Phase I. This is not meant to be an exhaustive list, only major items that customers have expressed as their reason to upgrade to 1.5.

 

• Added vCloud Director Database Support – Microsoft SQL Server and newer versions of Oracle. See release notes for further information
  
• Updated API with Call-Outs
  
• Fast Provisioning – This will not function without HW8 support and the completion of Phase III, and possible redesign of existing vCloud Organizations and/or Allocation Models may be needed.
  
• Advanced vShield Management
  
• Enhanced vApp Properties

Upgrading VMware vCloud Director Cells

The basic upgrade instructions are located in the VMware vCloud Installation Guides and are as simple as copying over the installer file and running it. However, the upgrade to vCloud Director 1.5 does update the database schema. You will also need to use the Cell Management tool to quiesce all the cells in the installation prior to upgrading.

Back Out Plan for the vCloud Director Cells

At this point it could be possible to roll back the upgrade, but this has NOT been tested as of yet and would need to be done before ANY new items are deployed in vCloud Director as those objects would be in the new database table structure. If a decision is to be made to roll back it needs to be done at this point so the database backup could be restored, and vCloud Director 1.5 uninstalled and 1.0.1 re-installed. Once you continue to move forward it becomes increasingly difficult to roll back due the additional functions in vShield Edge and the separate vShield Manager database. Additionally once users are back on the system and new objects are deployed there is no point of return at all (without object loss and much manual cleanup of existing vCenter objects no longer recognized by vCD, which still aren’t guaranteed to function properly).

There appears to be a few possible options to assist in rolling back the upgrade:

• Snapshot the vCloud Director Cells – We have always recommended that the cells are virtual machines. Therefore the snapshot functions in vCenter can be used.
  
• Backup the entire vCloud Database – It is also a recommendation that you backup the entire database for vCloud Director.
  

If both the snapshot and the backup are taken, one should be able to restore the database and revert/delete the snapshot. If items are deployed between the time of the backup for validation and testing there could be some items to clean up as those objects would not have been in the original backup.

Important Considerations and Requirements:

• Take a Full Backup of the vCloud Director Database
  
• Stop the services completely on vCenter Chargeback as only version 1.6.2 is compatible with vCloud Director 1.5. The 1.6.2 version is also NOT backward compatible with vCloud Director 1.0.1 so it is just easier to stop the services all together and upgrade this component last.
  
• Run required Red Hat patch updates prior to running the vCloud Director installer. The package dependencies have updates that seem to be required by vCloud Director 1.5.
  
  • DO NOT update kernel or other packages that would essentially bring the system to an unsupported version of RHEL.
    
  • In most cases you may want to update just the pre-requisite packages needed by vCloud Director 1.5 from the Installation guides.
    
  • Refer to the documentation, but there are new versions of RHEL that are supported just be sure not to patch beyond those.
    
• Consider lowering the TTL (Time to Life) on the DNS for the Load Balanced VIP’s for HTTP and Console Proxy a day or two prior to upgrading
  
  • Lowering these will allow clients to update their DNS cache quicker when resolving the portal name. Since we are re-directing the DNS name to temporary Maintenance pages and back a lower TTL will prevent the need for users to manually flush their DNS cache for updates.
    
• Redirect the DNS prior to upgrading to a custom Maintenance Page on a completely separate web server outside of the vCloud Director Cells. Ensure all users are now redirected to the Maintenance Page before shutting down cells.
  
• Install the Cell Management tool and follow the KB instructions for gracefully shutting down a Cell also shown below. Username can be any SYSTEM level user including an LDAP account. Perform this on ALL vCloud Director Cells. Even though user connections have been redirected, the Cells can still be performing tasks.
  
  • Log in to the target cell as root.
  • Navigate to $VCLOUD_HOME/bin/ 
    • (typically /opt/vmware/cloud-director/bin/).
      
  • Suspend the scheduler by typing the following command:
    • ./cell-management-tool -u -p -q true
      
  • View running tasks by typing the following command:
    • ./cell-management-tool -u -p -t 
      
  • When the task count reaches zero, shut down the Cell by typing the following command:
    • ./cell-management-tool -u -p -s
      
• Once all Cells are shut down and there is no connections to the database and you should have a Database Administrator back up the database
  
• You can run the vCloud Director installer
  
  • Version 1.5 creates a NEW separate directory called /opt/vmware/vcloud-director, but maintains the old directory (/opt/vmware/cloud-director) where most likely certificates.ks is located. There is no need to remove the old directory
    
  • Do not start the services yet
    
  • You will also need to update /etc/fstab for the new NFS mount point as it will be referring to the OLD transfer location under /opt/cloud-director. It is important to note that the installer will appear to MOVE the mount point to the new directory location but if the server is rebooted the cell will not start because FSTAB is still mounting the old location on each system boot.
    
  • On the first Cell upgraded you can also run the Database Upgrade script located at /opt/vmware/vcloud-director/bin/upgrade
    
    • NOTE: This is NOT done on the first server installed ONLY on the subsequent servers.
      
• Repeat the installer on all other Cells
  
  • DO NOT run the database upgrade script again
    
• Start the services on each Cell one at a time by REBOOTING one cell at a time to ensure the NFS shares are mounted.
  
  • Validate with /opt/vmware/vcloud-director/logs/cell.log that the services are coming back up
    
  • Also validate each Cell by connecting to their respective host HTTP ports and log into the portal
    
• DO NOT REDIRECT THE LOAD BALANCER nor allow users back onto the system yet
  

vCloud Director Connectivity to Hosts

VMware vCloud Director should update the host agent on all connected hosts. You should check the connected hosts in vCloud Director to ensure they are still showing available and “Prepared”. The vCloud Director 1.5 agent is different than the previous version and once installed should remain when you get to the ESXi Upgrade steps

Validation Steps to Perform After Cells Are Upgraded

The following two simple steps should be performed to ensure that functions are still working before moving to the next major step. These are basic but will ensure that there is a baseline of functionality as you move forward.

1. Deploy a new vApp
   
2. Create a NAT routed network (either a routed Organization Network or Fence a vApp that is deployed)
   

If either of these steps fail, stop here and troubleshoot the issue before moving on.

1. Rolling Back the Upgrade

If you decide you want to roll back at this point you can restore the database backup and revert/delete the snapshot. It should be understood that as you progress you will get closer to the point of no return for Phase I.

Upgrading the vShield Manager

Upgrading the vShield Manager utilizes the UPGRADE package file. Do NOT deploy a new vShield Manager or all the information in the local MySQL database containing deployed vShield Edge devices will be lost.

WARNING!!! – Do NOT attempt to redeploy the latest version of the vShield Edge Appliance. Removing the existing vShield Manager Appliance will break all connections, and management to any deployed vShield Edge devices resulting in errors. Once a vShield Manager is deployed you should upgrade it in place to the latest supported versions to ensure the local database remains intact. It may be possible to restore a configuration to a newly deployed vShield Manager using the FTP backup taken, but that has not been tested for this document.

Back Out Plan for vShield Edge Devices

The feeling among many folks is that once you have committed to upgrading the deployed vShield Edge devices you are past the point of no return. Trying to back out at this point could result in data loss and possible corruption of vShield Edge networks and devices. Again, once users are back on the system there is definitely no going back as new objects will all be created using the new products.

However there are two options for providing a method of rollback it is up to each individual to decide the best method

• Snapshot the virtual machine – This will require FT is disabled if you had it enabled
  
• FTP Backup – This is available in the Web UI
  

Procedures

• Snapshot the virtual machine OR
  
• Back up the local database using the Web UI FTP option under “Settings and Reports”
  

Figure 1 – vShield Manager Backup Option

• Upload the upgrade file using the “Updates” Section with “Upload Settings”
  

Figure 2 – vShield Manager Update Screen

• Once the file is uploaded you will see be able to select “Install”
  

Figure 3 – vShield Manager Update Install Screen

Figure 4 – vShield Manager Update Steps Screen

• The vShield Manager will go through reboots and when it comes back up it should show version 5.0 as the installed release. It will take a few moments for all the services to start before you can connect to the UI via the URL.
  

Figure 5 – vShield Manager Updated Build Screen

• vShield Manager 5.0 uses a security model different from the previous version. After the install is done you can still log in with the “Admin” account and password you used before, but you will see it is no longer connected to vCenter Server.
  
• You must wait 5-10 minutes per the release notes for vCloud Director to reconnect and populate the vCenter connection information.
  
  • This is NOT something you can change and has been identified as a possible feature request to allow vShield Manager and vCloud Director to use separate accounts. At this time vShield Manager will use the same credentials that are configured on the General Tab in the vCloud Director vCenter server general settings.
    
  • This is currently the expected behavior and has been verified by the vShield engineering team.
    
• Lastly, you MUST add the vShield 5.0 licenses into vCenter Server as they are not the same as the old version. Your upgraded vShield will show as Evaluation Mode until you do so.
  

Figure 6 – vShield Licensing Updates

• Validate everything is working by deploying a new NAT routed network to ensure a new vShield Edge is properly deployed before attempting to upgrade existing Devices.
  
• Here you must decide on a few things:
  
  • Upgrading all the deployed vShield Edge devices on Org Networks
    
    • If you decide to do this keep users off the system and proceed to 3.6.2
      
  • Upgrading all the deployed vShield Edge Devices on vApp Networks.
    
    • If you decide to do this as an administrator keep users off the system and proceed to 3.6.2
      
    • You can elect to allow vApp owners to do this on their own.
      

Validation Steps to Perform After vShield Manager is Upgraded

Do not perform this step until you have seen that the update from vCloud Director has populated the vCenter connection in vShield Manager. If you do not wait long enough you will get errors when trying to deploy or reset networks.

The following two simple steps should be performed to ensure that functions are still working before moving to the next major step. These are basic but will ensure that there is a baseline of functionality as you move forward.

1. Deploy a new vApp
   
2. Create a NAT routed network
   

Rolling Back vShield Manager

If you decide at this point to roll back the installation you can revert/delete the snapshot. You should not need to restore the database from FTP backup. Again, if you continue past this point you are essentially past the point of no return.

Upgrade Deployed vShield Edge Devices

Be sure to wait at least 5-10 minutes after upgrading the vShield Manager for it to report back to vCloud Director that it is updated. There is a documented issue in the release notes here:

http://www.vmware.com/support/vcd/doc/rel_notes_vcloud_director_15.html#installissues

System administrators will need to upgrade any Org Network-based vShield Edges. Org Admins can upgrade any vApp Network vShield Edges, OR the System admins can upgrade them. It is recommended that at least for the Org Networks, users remain off the system and the load balancer remain redirected to the maintenance page until the process is completed.

• Examine the version of a vShield Edge in vCenter and you will see the old build number
  

Figure 7 – vShield Edge Versions

• Upgrading the vShield Edges that are deployed is very simple. Just navigate to the Organization network under the Manage and Monitor section, right-click, and select “Reset Network”
  
  • If you watch vCenter, you will see a new virtual machine get deployed and the old one removed. The process is fairly quick and results in only a minimal outage to running virtual machines using this Edge device
    

     

Figure 8 – vShield Edge Updating Deployed Edge

 

Figure 9 – Ping Test During vShield Edge Update

• Examine the new Edge device for the new updated build number that should also match the build number of the vShield Manager.
  

 

Figure 10 – New vShield Edge Version

• Updating vApp networks can be achieved the same way; only you need to navigate to the vApp and select “Show Networking Details” to see the network vShield Edge. From there the process is the same, OR the vShield Edge will be updated if the vApp is completely powered off and then powered back on.
  
• YOU CAN NOW REDIRECT THE LOAD BALANCER BACK FROM THE MAINTENANCE PAGES AND ALLOW USERS TO ACCESS THE SYSTEM IF YOU CHOOSE TO HAVE THEM ON IT PRIOR TO CHARGEBACK BEING UPGRADED SINCE THE SERVICES ARE STILL DOWN.
  
• IF THE CUSTOMER WANTS TO UPGRADE THE ORACLE DATABASE AS WELL AT THIS POINT, DO NOT SEND USERS TO THE PORTAL UNTIL THAT IS COMPLETE.

Upgrading Oracle Database

Since vCloud Director supported on additional versions of Oracle 11g, customers can choose at this point to upgrade the Oracle database server as well. This will also require that the vCloud Cells are not connected to the database, resulting in an outage in most cases and you will need to use the maintenance pages again.

Migrating vCloud Director Database from Oracle to MS SQL Server

This is currently not supported and a SQL Server installation will require a clean database installation and a rerun of the vCloud Director configuration script. There may be unsupported ways of accomplishing this, but the customer would do this at their own risk.

Upgrading vCenter Chargeback

As with all other sections of this document, we will only address vCloud-specific issues when upgrading Chargeback to version 1.6.2. Follow all current documentation for upgrading vCenter Chargeback.

Rolling Back vCenter Chargeback

As with the previous components care should be take to allow a possible back out plan. There are two options for providing a method of rollback it is up to each individual to decide the best method

• Snapshot the virtual machine – This will require FT is disabled if you had it enabled
  
• Backup the Chargeback database
  

Considerations

• The installer will require that the previous version be uninstalled. You want to select “do NOT empty the database”.
  
• You can uninstall the plug-in, but reinstalling the plug-in will require the Service Account be Administrator in vCenter
  
• 1.6.2 is not backward compatible with vCloud Director 1.0.1 so this should be done last in this sequence.
  
• There will be database updates as well to support the new vCloud Director data.

Phase I Completion Verification Checklist