Well folks, there has been a lot going on for me recently as you can tell by the frequent postings, but this week was especially interesting for me. I had a unique opportunity to work with a customer on doing some high level architecture of vCloud running on top of Vblock. At the end of two grueling days of conversations I wanted to compile some of the findings from a vCloud Architecture perspective and let those of you out there draw some of your own conclusions. What I had specific to the customer will not be revealed and you will need to examine your specific Vblock configuration to see if and where you can make changes by working with VCE who is most likely setting it up for you. Also none of this is meant to be any kind of reference architecture, it is simply findings and possible suggestions to think about if you are doing a design. First let’s start with a few basic primers which was somewhat new to myself about what a Vblock configuration is. Some of this is probably posted in various places but I’m a simple guy so I like to explain it how I understand it. If any of the information in here is NOT accurate please contact me directly so I can update it properly. This is all my understanding of the components and configurations as it was explained to me, so please if it is incorrect help me change it, but do not shoot the messenger.
What is a Vblock?
From what I understand having worked with VCE this week, a Vblock to me is best described as a physically separate island of storage, compute, and networking that is then tied into the customer’s existing network infrastructure. Essentially a pre-configured “Block” of components purpose built and running VMware vSphere as the Datacenter Operating System. Now a few things that I learned are the following:
- Each Vblock is delivered with what;s called an “AMP Cluster” of 2 ESXi servers usually C-series rack mount units.
- The AMP cluster houses a vCenter VM that manages the AMP cluster itself as well as approximately 12 other Virtual Machines including UIM for managing the compute nodes and deploying ESXi on the UCS blades in the Vblock.
- Usually there is a second vCenter Virtual Machine installed on the AMP cluster to manage any UCS blades as vSphere clusters, but not always. – (This is actually a good thing for vCloud deployment if there is a separate one)
- There is also networking and storage management Virtual Machines in here as well along with SQL Virtual machines to house the vSphere databases.
- The AMP cluster hosts have 4 Physical NIC’s, 2 are used for ESX Management and two are used for VM traffic. In some cases the VM traffic NIC’s are physically cabled to a single Cisco 4k Switch which is not redundant, however VCE was kind enough to re cable them to the redundant 5K switches for us for a few reasons to be discussed later. Check with VCE on you config to see how your configuration is cabled as it can vary.
- Sometimes the Customer does not have access to the AMP vSphere servers or the vCenter managing them, but they do have access to the second vCenter. Again some designs vary here.
- UIM has the ability to provision Virtual Machines using the VIM API to the second vCenter. This is both good and bad and I will explain why. UIM to my knowledge does not yet interface with the vCloud API’s.
- In most cases VCE supports the entire Vblock itself from up through and including the vSphere stack. This to my knowledge includes updates and patching of vSphere itself.
- Typically the storage is configured for FAST and FASTCACHE to allow dynamic movement of workloads based on I/O from one type of disk to another, so vSphere just sees “Virtual LUNs” that are all basically the same.
- My understanding is Vblock is configured with the Nexus1000v option.
- Below is a general connectivity Diagram so you can see some what I am referring to. It is rather large I apologize and the full size PDF can be found here, but actual Virtual Machines are not shown.
vCloud on Vblock Design Considerations
Since we now have a basic understanding of the Vblock standard architecture we need to understand some of the possible design considerations to layer vCloud Director on top of this architecture. I will pose these in question and answer format as I think it is easier to follow and understand where you can diverge.
Question: Can I run the vCloud Management components, (vCD, VSM, Chargeback, and vCenter), in the AMP Cluster?
Answer: It depends, you must ask VCE if they are willing to allow this on a case by case basis. The better OPTION might be to carve out three to four UCS blades from the Vblock to act as the vCloud Management Cluster. This will separate the functions of the vCD stack from the VCE management stack completely. Also having three nodes for vCD management allows maximum high availability even during maintenance. Then you should have enough flexibility to run all the levels of management you need or want for vCD within the Vblock.
Question: If the storage tiering is Dynamic can Chargeback work with this model or are there other options?
Answer: Basically Chargeback has no idea of knowing what storage a vApp is sitting on with the virtual LUN’s. So in most cases you might need to pick an “Average Cost” per GB for the storage Chargeback models to cover everything from SSD to SATA. My understanding is you could disable this feature and configure traditional RAID groups tied to the disk types and then configure vDC’s based on the storage type to get your Gold, Silver, Bronze, and configure chargeback accordingly. Both options are viable from the discussions we had.
Question: Does VCE also support the vCloud Director component of the architecture?
Answer: Again, to my knowledge the answer today is no. They support the whole stack up through and including the vSphere installations. That may change in the future for now you need to check with VCE. The vCloud portion is most likely going to be supported by VMware unless VCE adds that to the stack.
Question: Does each Vblock equate to a separate vCloud instance?
Answer: I would say it depends, but most likely the answer is it does not. However depending on the Vblock vSphere Cluster configuration you could have multiple provider vDC’s in a single Vblock OR each Vblock itself could be a Provider vDC. In one case I worked on the Vblock itself had different UCS blades which resulted in 2 clusters and thus two provider VDC’s. Either way remember vCloud Director is Architected on a per physical site basis, so if one site has multiple Vblocks each would simply be resources available to vCloud Director. The same rules of Provider vDC’s applies that a Provider is created from the top level resource pool of a cluster so be sure to examine your individual Vblock and carve up the vSphere clusters accordingly based on vSphere cluster best practices. If the Vblock has a small number of UCS blades then it could very well be one cluster per Vblock, hence one Provider vDC per Vblock.
Question: Are my vCloud networking options “fixed”
Answer: In a sense they could be. My understanding is Vblock is usually delivered with the Nexus 100v option so you will be limited to Port Group Backed networking in vCloud Director. We just only use the Nexus for the External Network (SLAs etc) and the vDS for Org’s. This was suggested by Duncan Epping.
Question: I want to do scripted automation can I use both the vCloud API and the UIM API’s to deploy Virtual Machines?
Answer: The short answer is most likely this is not going to happen. If vCloud has been installed and is controlling the Virtual Machine and vApp constructs on the UCS blades, UIM has no knowledge of those because from what I have been told it talks only to the VIM API’s (vCenter) today. If you try to double up you will most likely mangle one or more VM’s and any VM’s created by the VIM API are not known by vCloud Director. You CAN however use UIM to do automated hardware level management on the storage, and deployment of vSphere ESXi hosts themselves. Try to think that UIM manages physical but if vCloud Director is in play it manages the Virtual side. In fact check out this article where Chad Sakac from EMC equates UIM to the hardware management partner to vCloud Director. There is also another good, more recent demo showing how UIM is for hardware then you move over to vCD to provision the vApp side of things.
“VMware vCloud Director is essentially a portal that sits on top of all of the vSphere resources and turns them into multi-tenant, subdivided virtual data centers. Unified Infrastructure Manager 2.0 sits on top of a Vblock infrastructure and creates a [scenario] where you don’t provision hardware, you provision service levels,”
Sakac said the combination of vBlock, UIM 2.0 and vCloud Director equates to a “cloud in a box”
Updates (2/22/11):
Well friends it seems this post has generated some more questions which I want to thank Richard Damoser from VMware for bringing to my attention. Here is the additions for more Q&A to ponder over
Question: What exactly is the impact of using Nexus1000v in vCD? I heard something about limitation of VLANs?
Answer: I think as was pointed out by Duncan Epping Nexus1Kv may have a limit of 512 VLAN’s but I am not 100% sure yet. The “Constraint” if you will is the fact as stated above you MUST use “Port Group Backed” pools which means a network administartor must create the Port groups before you can bring them into vCD.
Question: Will VCE allow deployment of vCD Management on ‘their’ management cluster?
Answer: This seems to be case by case and needs to be approved by VCE before just assuming you can do it. Sometimes to my knowledge customers do not have access to the AMP cluster at all.
Question: In regards to uptime monitoring what is already implemented in the vBlock? What has to be added?
Answer: I honestly do not know what is available via UIM for management. However, all vSphere monitoring tools apply since the underlying infrastructure is vSphere. CapacityIQ can be used in some cases with vCenter or new tools like Alive may soon have direct integration.
Question: Will VCE consider vCD dependancies when proposing updates to Cisco, EMC, and most importantly vSphere components?
Answer: I hope they will, but as best I can tell this is yet to be determined. I would think if they are involved in the architecture process and know vCD is in place they should be working with you to ensure compatibility based on published VMware documentation. The tricky part would be the AMP cluster and the UCS blade updates. The biggest suggestion is COMMUNICATION is key.
Conclusions:
Hopefully this provides some level of clarity to what you might have seen or heard. Again this is meant to only provide insight into what I saw from my perspective and it not meant to act as a reference architecture. Please refer to all the vCloud documentation and the current reference architecture for more details. Use this information simply to help you think about the other considerations that are there both technically and operationally when it comes to architecture vCloud on Vblock. Please feel free to email me more questions so that I might be able to add to this article or post constructive comments. As always engage VCE and VMware to help provide the best architecture for your requirements, it is what we do best.
Updates 4/29/11):
Myself and VCE are currently working on a Whitepaper/Solution Brief to address much of this and take into account what VCE can and will support. However, much of the items here are being considered so please stay tuned for more information in the next month or so. The Brief will be based on vCloud Director 1.0.1 and the current 1.6 Reference Architecture.