In the course of building my lab for my Nicira NVP role, I needed to install various hypervisors, in particular Linux KVM so I could install and test the Nicira NVP setup. Of course I did not want to do this on bare metal and I have no intention of trying to set up OpenStack or CloudStack just yet, I can make do with the hypervisors only. I will do some other blog posts on the importance of the Cloud Management System to the Nicira NVP components. For the purposes of this post I wanted to share the documentation I collected on just what is needed to get KVM on CentOS 6.3 up and running specifically for use with Nicira NVP.
Step 1 – Configure the Virtual Machine Shell
This was something I learned the hard way. With vSphere 5.1 you cannot create a new VM in the Windows vSphere client and allow the virtualization features of the processor to be presented. This check box option as best I can tell is only available in the new vSphere Web Client. So you will want to use the web client to create the shell not the windows client.
You can see the Virtualization option below:
You will also notice this option will only be available if you select Compatibility of ESXi 5.1 and later VM Version 9. GO ahead and select CentOS (64-Bit) as the guest operating system for now. Once you have set up your Virtual Machine shell with the memory and other options you want, you can start your OS install. My basic shell set up:
- 4 vCPU
- 6GB RAM
- 20GB Disk
- 3 NIC’s (Management, NVP Transport, Storage)
Step 2 – Install CentOS with Virtualization Features
This is pretty easy. Fire up the CentOS installer and selection “Virtual Host”. You also want to customize the install and add in the “Virtualization Tools”. Once the install is completed the real work can begin. You can also do the “Minimal” install from the smaller ISO file, but then you will need to add all the virtualization group components which is also easy enough using the following command:
# yum groupinstall "Virtualization*"
Step 3 – Install Other Packages, Updates, and Tools
At this point you may want to add and configure a few things so you can clone this VM and be ready for further customization. Below are some of the updates, packages, and edits you may want to do on this host.
# yum install dejavu-lgc-sans-fonts xorg-x11-xauth openssh-askpass make gcc kernel-headers # yum update # reboot
NOTE: – When you SSH use -X to enable X11 redirection to your client for running virt-manager, and you will need an X11 setup on your Mac (XQuartz), or Windows.
Copy the Open vSwitch files to the Host and install them:
# tar zxvf nvp-ovs-*.gzyum # install kmod-openvswitch-*.rpm openvswitch-*.rpm
Personally in my lab I also disabled SELinux and IPTables as I was not sure what they would do to KVM.
Edit /etc/selinux/config and set SELINUX=disabled
# chkconfig iptables off # chkconfig ip6tables of # service iptables stop # service ip6tables stop
Edit the udev rules:
vi /etc/udev/rules.d/70-persistent-net.rules delete the lines that begin with SUBSYSTEM=="net"
Edit the ethernet device files:
vi /etc/sysconfig/network-scripts/ifcfg-eth0 vi /etc/sysconfig/network-scripts/ifcfg-eth1 ..... comment out the line that starts with HWADDR=
Once you have updated and rebooted follow the standard operating procedure to install VMware Tools on this Linux Virtual Machine, and at this point you can shutdown and clone the Virtual Machine to a Template and re-clone to a new Virtual Machine.
Step 4 – Configure Linux Networking and OVS Bridges
Finally configure your Networking and Name Servers. For networking you will need to set up Linux Bridges for each Interface for later use by KVM, and Nicira NVP Open vSwitch. You’re files should look similar to the examples below but first create new Bridge Interfaces:
Create new bridge interface configuration files:
# cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-br0 # cp /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/sysconfig/network-scripts/ifcfg-br1 # cp /etc/sysconfig/network-scripts/ifcfg-eth2 /etc/sysconfig/network-scripts/ifcfg-br2
| Example Configuration for ifcfg-eth0:DEVICE=”eth0″ ONBOOT=”yes” NAME=”System eth0″ |
Example Configuration for ifcfg-eth1:DEVICE=”eth1″ ONBOOT=”yes” NAME=”System eth1″ |
Example Configuration for ifcfg-eth2:DEVICE=”eth2″ ONBOOT=”yes” NAME=”System eth2″ |
| Example Configuration for ifcfg-br0:DEVICE=”br0″ TYPE=Bridge BOOTPROTO=”static” ONBOOT=”yes” IPADDR=<Management IP> NETMASK=<netmask> GATEWAY=<gateway> NAME=”System breth0″ |
Example Configuration for ifcfg-br1:DEVICE=”br1″ TYPE=Bridge BOOTPROTO=”static” ONBOOT=”yes” IPADDR=<Transport IP> NETMASK=<netmask> NAME=”System breth1″ |
Example Configuration for ifcfg-br2:DEVICE=”br2″ TYPE=Bridge BOOTPROTO=”static” ONBOOT=”yes” IPADDR=<Storage IP> NETMASK=<netmask> NAME=”System breth2″ |
Create Open vSwitch Bridges for Each Interface
# ovs-vsctl add-br br0 # ovs-vsctl br-set-external-id br0 bridge-id br0 # ovs-vsctl set Bridge br0 fail-mode=standalone # ovs-vsctl add-port br0 eth0
# ovs-vsctl add-br br1 # ovs-vsctl br-set-external-id br1 bridge-id br1 # ovs-vsctl set Bridge br1 fail-mode=standaloneo # ovs-vsctl add-port br1 eth1
# ovs-vsctl add-br br2 # ovs-vsctl br-set-external-id br2 bridge-id br2 # ovs-vsctl set Bridge br2 fail-mode=standalone # ovs-vsctl add-port br2 eth2
Restart The Network Services
# service network restart
Step 5 – Edit /etc/libvirt/qemu.conf
This is required for any version of libvirt that is 0.9.10 or lower and for the case of CentOS 6.3 the version of libvirt is 0.9.10
Find and Uncomment:
cgroup_controllers = [ "cpu", "devices", "memory" ]
Find and Uncomment, as well as add the reference to “/dev/net/tun”
cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc", "/dev/hpet", "/dev/net/tun" ]
Find, Uncomment and set:
clear_emulator_capabilities=0
Find and uncomment:
user = "root" group = "root"
Restart the libvirt service:
# service libvirtd restart
Step 6 – Create Generic Control Scripts for libvirt 0.9.10 and Lower
Like the above process these scripts are used ny libvirt to connect the Virtual Machines to the Open vSwitch on power up and power down. This is only needed for libvirt 0.9.10 and lower as well and that is the case for CentOS 6.3
Create /etc/ovs-ifup-generic and make the file executable:
#!/bin/sh if [ $# -ne 3 ]; then echo "usage: DEVICE IFACE_ID ATTACHED_MAC" exit 1 fi /sbin/ifconfig $1 0.0.0.0 up ovs-vsctl add-port br-int $1 -- \ set Interface $1 external-ids:iface-id=$2 -- \ set Interface $1 external-ids:attached-mac=$3 -- \ set Interface $1 external-ids:iface-status=active
For convienence I have also supplied a version of the file here: [s3file s3url=”ovs-ifup-generic.txt” ]Download File[/s3file]
Simply download it and copy it to /etc/ovs-ifdown-generic and make it executable
Create /etc/ovs-ifdown-generic and make the file executable:
#!/bin/sh /sbin/ifconfig $1 0.0.0.0 down ovs-vsctl del-port $1
For convienence I have also supplied a version of the file here: [s3file s3url=”ovs-ifdown-generic.txt” ]Download File[/s3file]
Simply download it and copy it to /etc/ovs-ifdown-generic and make it executable
Replace script links with new links:
# rm /etc/qemu-if* # ln -s /etc/ovs-ifup-generic /etc/qemu-ifup # ln -s /etc/ovs-ifdown-generic /etc/qemu-ifdown
Step 7 – Create and Deploy Virtual Machines
Now you can create virtual machines with the virsh commands or using SSH with the -X switch and using virt-manager. Either way once you create them you need to edit the XML files networking section in order to use these new scripts and connect them to the Open vSwitch. In most cases this is where the Cloud Management System comes into play, but I will at least give the changes to the files below.
Setup Per VM Control Scrips To connect NIC’s:
The actual location and file strucure is up to you but you will need to know this to edit the Virtual Machine’s XML file
# vi /var/lib/libvirt/images/[DOMAIN]/[DOMAIN]-interfaces Edit the File with these lines where [DOMAIN]=The Virtual Machine Name #!/bin/sh /etc/ovs-ifup-generic $1 [DOMAIN]-iface1
Edit the Domain (Virtual Machine’s) XML file for the BOLD sections:
# virshedit [DOMAIN]
......Snipped /usr/libexec/qemu-kvm
<target dev='[DOMAIN]/>
<model type='virtio'/>
</interface> <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/> </devices> .......Snipped
Conclusion
You can see that setting up KVM on CentOS 6.3 is not that trivial and for sure the manual insertions of the script files and other items needed into the Virtual Machine’s XML file are clearly where the Cloud Management System comes into play. You would not do this all manually, except maybe like me in a lab for a couple Virtual Machines to test. As I continue to move forward I will be adding another Article on building XENServer on vSphere 5.1 along with some posts on Nicira NVP itself from my own perspective with maybe a few demo videos using my lab setup. I may put together a full lab diagram now that it is all working with all the primary Nicira NVP components.