I realized yesterday in talking to a friend and colleague of mine that there is a lot of confusion about load balancing vCloud Director cells. I had written a detailed post about considerations and many of these were also incorporated into the new vCAT 2.0. However it seems there is still something lost in translation so I wanted to break it down to very some very simple requirements that are easy to follow. Below is the requirements needed to get load balancing working. Assuming you have a network admin to work with this will NOT be the instructions on configuring a specific load balancer, but these are the requirements you can give to your engineer. You can change the hostnames of course these are just for illustration and to get the point across.
Let’s start first by defining the DNS and IP address that are needed to make the configuration work, then we will go into the other items you need to get the network team rolling. You need to have a little understanding of SSL certificates and client termination to understand this as well.
Be sure you setup the shared transfer storage per the documentation. A key component of the multi-cell setup is that you have the transfer directory mapped to a shared location on each cell. This is done by editing FSTAB to alter the mount point on the cells for /opt/vmware/vcloud-director/data/transfer.
IP Addresses and Host Names
vCloud Director Cell HTTP DNS:
host1.companyname.com = HTTP IP address
host2.companyname.com = HTTP IP address
vCloud Director Cell Remote Console DNS:
host1-con.companyname.com = Console IP address
host2-con.companyname.com = Console IP address
Load Balancer DNS Entries
vcloud.companyname.com = HTTP VIP IP address
vcloud-con.companyname.com = Console VIP IP address
Now that we have defined the basic IP addresses we need, we need to also make sure the certificates are setup properly. Assuming you want to use signed certificates here is what I have found works best for assigning the certificates.
HTTP Certificates (With SSL Offload for HTTP):
For the individual Cells you want to issue a certificate that MATCHES the hostname above. This will be used by the load balancer to connect via SSL to the hosts in the pool. Also this will allow people to connect directly to a cell without a certificate error. Then you can obtain a certificate for the Load Balancer VIP address to install directly onto the load balancer. This will be the secure connection the clients use when connecting through the load balancer. This setup ensures client to load balancer and load balancer to cell is encrypted.
Console Certificates (THIS CANNOT USE SSL OFFLOAD!!!):
This is where people break things. The Console Proxy port although on PORT 443 is NOT in fact an HTTPS web connection. It is a pure socket connection. For this reason you cannot do SSL offload and this VIP/pool needs to be passthrough to work. Therefore to avoid SSL certificate errors you do NOT want to issue the console proxy certificates to the hostnames listed above. You actually want BOTH console proxy ports to use the SAME certificate hostname of ‘vcloud-con.companyname.com‘. This way when passthrough happens the clients will see the same certificate name presented to them.
HTTP Certificates (WITHOUT SSL Offload for HTTP):
If for some reason you cannot or do not want to terminate client SSL on the load balancer and leverage SSL offload, then you can setup the HTTP certificates like the console proxy and do pass through. Just like with the console proxy though you will need to install a certificate with the SAME hostname if you do not wat the clients to get browser errors. However if you browse to a server directly you will get the name mismatch error.
Important NOTE: These must be named EXACTLY as ‘http’ and ‘consoleproxy’, (Without the quotes), when you create the requests or if you use self signed. vCloud Director is coded to only look for those certificate names. Shout out to Rajeev Karamchedu for pointing that out.
vCloud Director Configuration
Lastly, for all this to work there is a requirement to update the administration settings for the Public Address URL’s. This can be found under the Administration section. However each of the three has a different syntax in order to work properly, and you can see the addresses are based on the above original configurations. These need to be exact and you will notice the console proxy does NOT use “HTTP”
VCD Public URL = (Notice HTTPS)
VCD Public Console Proxy Address = vcloud-con.companyname.com (NO HTTPS!!)
VCD Public REST API Base URL = (Notice NO /cloud)
These are EXTREMELY important because they will redirect the client requests always back to these addresses. If you do not set these you will remain on the same server and your console proxy connection will also be directed to that same server and not be load balanced. Bottom line is if you have a load balancer you WANT to set these URL’s.
Final Load Balancer Requirements
There is a few other requirements most load balancer configurations should have and that is the Health Monitors and SSL redirects. For the health monitors there is one that will check the HTTP side of things, but it is important to also monitor the Console Proxy for various reasons. The two services are not “Dependent” on each other and you COULD have Cells that are in a load balance pool JUST for console connections in a high traffic environment. Then you may have cells that only handle HTTP/API traffic. This is completely doable from a design use case so below are the two monitor addresses. The key below is each one is looking to the DIFFERENT configured IP addresses
HTTP Health Monitor = http://<Cell_Hostname>/cloud/server_status
Console Proxy Health Monitor = http:///sdk/vimServiceVersions.xml
I am pretty sure the vCAT document mentions this but it is worth repeating that you want to set SSL persistance as well. This will ensure the Portal connections and console proxy connections are at least persistant for the duration of the SSL connection. Source IP persistance can be used but, it may not expire as quickly as SSL persistance.
Lastly you will want your load balancer to redirect HTTP connections to both VIP pools to HTTPS. This is just good load balancer web practice than if the VIP gets an HTTP request it redirects the client to the HTTPS version.
The SSL Offload method you outline above isn’t really offload if you are decrypting and then re-encrypting the traffic on the LB is it? The cell will still need to decrypt this traffic so you haven’t “offloaded” anything.. Is there no way to use just http on the internal side?
Tim – You are right in a sense it is not “TRUE” offload since there is still SSL between the LB and the Cells. Currently, NO there is no way to just use HTTP on the internal side, this has and continues to be a popular open feature request. I hope this will be addressed at some point in a future release.
I’m attempting to configure a load balancer for the consoleproxy service and was hoping you could clarify something for me. For the consoleproxy certificate, you say to use the same certificate name (the VIP name), which to me implies generating different certs for each cell but with the same CN. However, that doesn’t seem to be working; it still results in certificate errors. Does it really need to be the exact same certificate imported into each cell? Thanks for any advice you can provide…
Yep, using the same cert+private key on each cell seemed to do it. I generated the certificates.ks file and .csr on one cell, imported the CA-signed cert, then used keytool to export the consoleproxy cert+key to a PKCS12 certstore, then imported the cert+key to the certificates.ks file on the second cell. Re-ran the configure script and now it works.
For future reference, here is the keytool command to export the key from the first cell:
/opt/vmware/vcloud-director/jre/bin/keytool -v -importkeystore -srckeystore certificates.ks -srcstoretype JCEKS -srcalias consoleproxy -destkeystore certs.p12 -deststoretype PKCS12
And here is the command to import the key on all other cells:
/opt/vmware/vcloud-director/jre/bin/keytool -v -importkeystore -srckeystore certs.p12 -srcstoretype PKCS12 -srcalias consoleproxy -destkeystore certificates.ks -deststoretype JCEKS
Is there any way to protect the VMRC port from malicious traffic (e.g. attempts to exploit or crash the service that’s listening on that port)?
There’s a range of IDS products that can sit in front of the http port and filter out exploits and hacks but they all expect normal http traffic so won’t work for VMRC.
That’s a good question, but I am not sure the answer.