====== KVM_Setup_on_Debian_Lenny ====== All the following is done on a base install of Debian Lenny running a x64 kernel (2.6.26-2-amd64). In the case below it was done on a remote server with only ssh access to the host. LVM was already setup on the Debian Lenny Host and is used to give a LV to the VMs. ====== Setup of KVM and Libvirt on Debian Lenny Host ====== apt-get install kvm libvirt-bin virtinst #libvirt allows easy management of KVM, ensuring VMs start upon boot ====== (Default) Network Config ====== The default network config that comes out of the box with KVM gives a private IP address to each VM via local DHCP & DNS server (dnsmasq). Each VM can then NAT out through the host to get internet access. There is an issue with dnsmasq and possibly a bug with debian which is overcome below by preventing dnsmasq to start on boot of the host and allowing libvirt instead of control dnsmasq. #Type "virsh" to enter the virsh console: virsh virsh # //You are now in the virsh console virsh # nodeinfo //Shows info on the Host virsh # list --all //Shows all VMs. (Same as going ~# ls /etc/libvirt/qemu/) virsh # net-list --all //Shows all the network config files. (Same as going ~# ls /etc/libvirt/qemu/networks/) virsh # net-edit default //It is BEST to edit network config files using vish. //If you edit /etc/libvirt/qemu/networks/default.xml manually, it won't update without a net-undefine and net-define. virsh # quit #DHCP of the VMs is provided by dnsmasq. We need to stop the dnsmasq service from starting itself and let libvirt control libvirt instead /etc/init.d/dnsmasq stop sysv-rc-conf (apt-get install sysv-rc-conf if its not installed) #untick dnsmasq for all runlevels virsh virsh # net-autostart default virsh # net-start default virsh # quit ======Setup of Debian Lenny VM====== wget http://ftp.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/mini.iso lvcreate -n deb01 --size 20g vg0 //For file based disk: dd if=/dev/zero of=/kvm/deb01.img bs=1024k count=20000 virt-install -d --name=deb01 --ram 512 --disk path=/dev/vg0/deb01 --network network=default --vnc --accelerate --cdrom /root/mini.iso # --accelerate IS required, otherwise it will be qemu and virtio won't work later when we config it manually. #After virt-install, you will see (virt-viewer:3501): Gtk-WARNING **: cannot open display: #Domain installation still in progress. You can reconnect to the console to complete the installation process. #Connect with vncviewer. You may have to ssh to the server and port forward port 5901. #netstat -tap //This will show the ports open and if they are bound to 127.0.0.1 #Complete the install. Debian will shutdown the VM after installation virsh virsh # list --all virsh # start deb01 #Connect with VNCviewer and apt-get install ssh ====== Benchmark ====== I ran many dd tests and hdparm however the results seemed too good due to KVM caches etc. As a result the following phoronix benchmark tests proved to work very well for comparing IO operations. vi /etc/apt/sources.list #Add the following line: deb http://www.phoronix-test-suite.com/releases/repo pts.debian/ apt-get update apt-get upgrade apt-get install phoronix-test-suite phoronix-test-suite benchmark apache //Scored Average: 8115.92 Requests Per Second phoronix-test-suite benchmark compress-gzip //Scored Average: 20.61 Seconds ====== Change from IDE to Virtio ====== I was mainly interested in IO performance of VMs. I've noticed a little iowait on my current Xen setup and wanted to see how KVM would work. virsh virsh # shutdown deb01 virsh # edit deb01 #change to: #add in under the network interface to look like: :wq ===== Mount LVM VM Disk ===== See: http://wiki.kartbuilding.net/index.php/Mount_kvm_file_based_image_(disk.img)_on_host_computer kpartx -av /dev/vg0/deb01 mount /dev/mapper/vg0-deb01p1 /mnt/ vi /mnt/boot/grub/menu.lst #change /boot/vmlinuz-2.6.26-2-amd64 root=/dev/hda1 ro quiet to: /boot/vmlinuz-2.6.26-2-amd64 root=/dev/vda1 ro vi /mnt/etc/fstab #Change hda* to vda* umount /mnt/ kpartx -dv /dev/vg0/deb01 virsh start deb01 ====== Custom Network Config ====== So my KVM host IP is x.x.x.16 My hosting provider also allow me 3 additional IPs: x.x.x.35, x.x.x.36 and x.x.x.61 I am using this config below AND the default KVM Network config at the top of this page. vi /etc/network/interfaces auto br0 iface br0 inet static address x.x.x.16 netmask 255.255.255.255 bridge_stp off bridge_fd 0 pre-up brctl addbr br0 #pre-up brctl addif br0 vnet3 //If VMs are running and init.d/networking restart done,this line will have to be done manually up ip route add x.x.x.35 dev br0 up ip route add x.x.x.36 dev br0 up ip route add x.x.x.61 dev br0 post-down brctl delbr br0 //Allows for smooth use of /etc/init.d/networking restart ifup br0 #check with "route" to see the entry. The virt-install config line is now: virt-install -d --name=deb03 --ram 512 --disk path=/dev/vg0/deb03 --network bridge=br0 --vnc --accelerate --cdrom /root/mini.iso The resulting xml network config is: virsh edit deb03 When completing the installation after virt-install using VNC, DHCP fails (of course) and Manual Network Configuration has to be done. Supply the fixed IP (x.x.x.35), the Default Gateway (Host IP x.x.x.16) and DNS server IP. That will then allow the netinstall to complete. ===== Custom Iptables Firewall with libvirt ===== As I wanted to use libvirt to create a default network with dhcp, I had to work around the fact that libvirt wanted to append its own iptables rules which were a little restrictive. I wanted VMs on the Private LAN to see other VMs on br0 and visa versa. As libvirt appended iptables rules, there was an issue. I tried removing the however libvirt still added its own iptables rules. ==== Firewall Script ==== vi /etc/init.d/firewall #!/bin/sh #This gets called by /etc/rc.local IPTABLES=/sbin/iptables EXTBR=br0 INTBR=virbr0 PRIVATE=192.168.1.0/24 $IPTABLES -F INPUT $IPTABLES -F OUTPUT $IPTABLES -F FORWARD $IPTABLES -F POSTROUTING -t nat $IPTABLES -F PREROUTING -t nat ##################### # DNS and DHCP ##################### $IPTABLES -A INPUT -i $INTBR -p udp -m udp --dport 53 -j ACCEPT $IPTABLES -A INPUT -i $INTBR -p tcp -m tcp --dport 53 -j ACCEPT $IPTABLES -A INPUT -i $INTBR -p udp -m udp --dport 67 -j ACCEPT $IPTABLES -A INPUT -i $INTBR -p tcp -m tcp --dport 67 -j ACCEPT #################### # FORWARDS ################### $IPTABLES -A FORWARD -d $PRIVATE -o $INTBR -m state --state RELATED,ESTABLISHED -j ACCEPT $IPTABLES -A FORWARD -s $PRIVATE -i $INTBR -j ACCEPT $IPTABLES -A FORWARD -i $INTBR -o $INTBR -j ACCEPT $IPTABLES -A FORWARD -i $EXTBR -o $EXTBR -j ACCEPT ################### # NATTING ################### $IPTABLES -t nat -A POSTROUTING ! -d $PRIVATE -s $PRIVATE -j MASQUERADE $IPTABLES -t nat -A POSTROUTING ! -s $PRIVATE -d $PRIVATE -j MASQUERADE ################### # PORT FORWARDING (Remote Desktop) ################### $IPTABLES -t nat -A PREROUTING -p tcp --dport 8883 -j DNAT --to 192.168.1.42:3389 $IPTABLES -t nat -A PREROUTING -p tcp --dport 8884 -j DNAT --to 192.168.1.152:3389 ################### # BLOCKING ################### $IPTABLES -A FORWARD -j REJECT --reject-with icmp-port-unreachable ==== Control the Firewall ==== We need to let libvirt start the default network and then run our firewall. Note: this is much less than ideal. The alternative would be not to use a libvirt network config, create a second bridge, do the dnsmasq manually and iptable rules. vi /etc/rc.local #add in the following while [[|`ps -e | grep -c libvirtd` -lt 1 ]]; do sleep 1 done sleep 10 /etc/init.d/firewall exit 0 The firewall can also be called manually anytime by going /etc/init.d/firewall Ref: http://efreedom.com/Question/2-170079/Forwarding-Ports-Guests-Libvirt-KVM ====== Munin Plugins for Libvirt / KVM ====== Debian squeeze has plugins for libvirt available via apt. Lenny does not however. The following worked ok for me on Lenny: //taken from http://honk.sigxcpu.org/projects/libvirt/ wget http://honk.sigxcpu.org/projects/libvirt/monitor/releases/munin-libvirt-plugins-0.0.6.tar.gz gunzip munin-libvirt-plugins-0.0.6.tar.gz tar -xvf munin-libvirt-plugins-0.0.6.tar #explore the directory and move the libvirt-cpu -ifstat etc. into /usr/share/munin/plugins and chmod 755 and then ln -s to /etc/munin/plugins/ ====== Setup of Windows 7 VM ====== lvcreate -n win01 --size 20g vg0 virt-install -d --name=win01 --ram 512 --disk path=/dev/vg0/win01 --network network=default --vnc --accelerate --cdrom /root/en_windows_7.iso ===== Correct Mouse Pointer issue with VNC ===== virsh shutdown win01 virsh edit win01 //add in the following line above the mouse config: //afterwards it will look like: Refs: http://wiki.libvirt.org/page/UbuntuKVMWalkthrough ====== IPv6 Config on Host ====== So my hosting company gave me a ipv6 /64 subnet. They also gave me a default gateway to which a route had to be added. Also debian lenny required "pre-up modprobe ipv6" to work correctly. * IPv6 Subnet: 2a01:4f8:100:xxx1:: * IPv6 Host IP: 2a01:4f8:100:xxx1::1 * IPv6 Default Gateway: 2a01:4f8:100:xxx0::1 iface eth0 inet6 static address 2a01:4f8:100:xxx1::1 netmask 64 gateway 2a01:4f8:100:xxx0::1 pre-up modprobe ipv6 pre-up ip -6 route add 2a01:4f8:100:xxx0::1 dev eth0