Saturday, December 31, 2011

Ubuntu 10.10 KVM Server Managed by ConVirt

Project Successful, but abandoned in lieu of virt-manager VMM


Findings:  I will abandon use of ConVirt 2.0 in favor of virt-manager.
ConVirt is not for the faint at heart and imho its limitations and challenges overshadow its advantages at this point in time.  Keep watching, though, maybe someday, if they stick with it they will have a world class Opensource Hypervisor!



Pro's of virt-manager over ConVirt 2.0
  • Cleaner interface
  • Lower footprint and easier setup
  • Does everything a small shop needs
  • More options on disk management
  • No need to have a VM running constantly just for running the interface
  • Much more active development
  • Simple server configuration with minimal extra packets and processes running at host level
Cons of virt-manager
  • Need to run from within linux (or Mac via ssh -X with virt-manager installed on server)
    • if you have Linux or Mac anyway, not a big deal - or you could create a VM dedicated to running virt-manager, which is in effect what ConVirt required anyway.
    • you could set it up to tunnel in remotely and run virt-manager via remote x session
  • no gui network setup tools, need to be done in config files or via commands --- for now!
    • simple bridging or NATing is easy though
Pro's of ConVirt 2.0
  • Web interface - log in from anywhere
  • Better statistics
  • live-migration - yes but virt-manager also does this handily now
Cons of ConVirt 2.0
  • No longer seems to be focus on Convirture... now Enterprise Cloud ... will product fade away?
  • Complicated setup
  • Disorganized GUI
  • Few extra real features for the fuss
  • Requires root login to all managed servers



Overview of Tasks
  1. Prepare Server
    • Install Ubuntu 10.10
    • Install ConVirt tools
    • 1.5 - prepare PC to host virtual machines via KVM
  2. Prepare Machine for receiving ConVirt 2.0
    • Use existing computer or virtual machine or prepare virtual machine manually
  3. Install ConVirt 2.0 (CMS)on a separate machine
    • Install ConVirt 2.0 Framework on the above machine, specifically this refers to installing the Convirt Management Server on the machine used to manage the Convirt enabled managed servers
    • 3.5 - setup Convirt to connect to managed servers
  4. Exploring vir-manager VMM as alternative
1)  Prepare Server
Starting with Ubuntu Server 64 bit 10.10.  Roughly following this guide:  https://help.ubuntu.com/community/KVM

  • Install Ubuntu 64bit Server 10.10 using appropriate options and the following:
    • choose to apply security updates automatically
    • choose software to install: 
      • OpenSSH server
      • Virtual Machine host
  • Install ConVirt tools http://www.convirture.com/wiki/index.php?title=Convirt2_Installation#Preparing_Managed_Servers
    • installed packages the document suggested I require
      • sudo apt-get install ssh kvm socat dnsmasq umo-utilities lvm2 expect
      • many but not all of these packets were not yet installed
    • run the convirt-tool script which "creates appropriate public bridges, required scripts and writes a summary of its operations to the /var/cache/convirt/server_info file"
      • enable root account https://help.ubuntu.com/community/RootSudo
        • sudo passwd root
        • (could I have done this using a persistent root login via [sudo -i] command?)
      • used wget to download tarball and unpacked it (tar -xzf convirture-tools-2.0.1.tar.gz)
      • message regarding using convirt tools given with command ./convirt-tool -h
        • shows qemu-kvm-0.12.5, Ubuntu 10.10, Kerneel 2.6.34-11, etc
      • ./convirt-tool install_dependencies
        • checks dependencies
        • installs kpartx & python-pexpect
      • brctl show
        • shows bridges, in my case virbr0 is currently setup, of course I will want to setup a br0 or the like which is linked to eth0
      • ./convirt-tool setup
        • this ran through the setup very fast, but 
        • when I ran the brctl show command again I see 
          • br0 attached to eth0
          • virbr0
      • nano /etc/network/interfaces
        • wanted to see the network setup out of curiosity, which looks much like the one I setup on my PC:
        • (loopback stuff, then:)
        • auto eth0
        • iface eth0 inet manual
        •  
        • auto br0
        • iface br0 inet dhcp
          • bridge_ports eth0
          • bridge_fd 0
          • bridge_stp off
          • bridge_maxwait 0
      • ifconfig
        • shows br0 with ip address of server and eth0 with no ip address... as expected since it connects through the bridge, also shows the virbr0
        • nothing to do here
      • nano /etc/libvirt/qemu/networks/default.xml
        • shows the virtbr0 interface 
        • nothing to do here
    • Adding to the CMS - done after setting up the CMS as a virtual machine
      • see below
    • VNC setup
      • done, see below
1.5) Prepare PC to also host Virtual machines under KVM
 I decided to also prepare my main personal computer to host virtual machines and be controlled by ConVirt 2.0, that way I can move VM's from the server to my PC and vice-versa.  Generally followed: https://help.ubuntu.com/community/KVM/Installation  
  • sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
    • This installed core packages needed for KVM
  • sudo apt-get install virt-viewer
    • so I can view virtual machine instances on computer outside of ConVirt
  • log out, then back in to affect new user group changes
  • virsh -c qemu:///system list
    • this command shows virtual machines (currently none), but also verifies the install went well
  • sudo apt-get install virt-manager
    • installs GUI tool to manage virtual machines (outside of ConVirt)
    • Works well, setup a 
To Allow Bridging to my Computer, needed to do the following based on:  http://doc.ubuntu.com/ubuntu/serverguide/C/network-configuration.html#bridging
  • gksudo gedit /etc/network/interfaces
    • allows editing of interfaces to define bridge interface
    • Added the following lines to the file
      • auto eth0
      • iface eth0 inet manual
      • #this line was changed from ...inet dhcp

      • auto br0
      • iface br0 inet dhcp
        • bridge_ports eth0
        • bridge_stp off
        • bridge_fd 0
        • bridge_maxwait 0
  • sudo /etc/init.d/networking restart 
  • This is not working yet!!!  found the following:  https://help.ubuntu.com/community/KVM/Networking   *** most helpful article found ****
    • Bridged networking does not work by default, so need to do some further setup
    • sudo apt-get install libcap2-bin
    • sudo setcap cap_net_admin=ei /usr/bin/qemu-system-x86_64   (did not work, hmmm)
    • sudo setcap cap_net_admin=ei /usr/bin/qemu
    • gksudo gedit /etc/security/capability.conf
      • add line
        • cap_net_admin    chenier
  • This was not successful, though bridge br0 is now working properly, the network is not working properly for the main computer.  --- br0 and eth0 are showing same ip address and MAC address. also, the computer says the wired network device is not managed.  Also, running /etc/init.d/networking restart gives message "...is deprecated because it may not enable again some interfaces... *Reconfiguring network interfaces... RTNETLINK answers: No such process; ssh stop/waiting; ssh start/running, process 3083; ssh stop/waiting; ssh start/running, process 3199"
  • Still problems, tried the cap_net 64 command and it did not throw an error this time
  • also changed /etc/network/interfaces line 
    • from..... iface eth0 inet dhcp
    • to....       iface eth0 inet manual
  • Now when rebooting everything works fine, with the exception that the gui networking device manager does not see or control eth0... is this now a function of capnet???  Actually, probably part of the following two points.
  • no prob, the command ifconfig shows proper connetion of br0 and any new VM I create connects to my DHCP server to get its IP address, so it appears all is well.  
  • Also, now ifconfig shows no IP address for my eth0, just for my br0, but I usnderstand that this is normal and that eth0 automatically gets its traffic through br0.
  • I am a bit concerned about the SSH information given when restarting networking (sudo /etc/init.d/networking restart) still gives the messages "ssh stop/waiting; ssh start/running, process 3083; ssh stop/waiting; ssh start/running, process 3199".  Maybe this is part of the RTNETLINK process and is normal.  I will want to verify this on the server install.
  • In this guide: https://help.ubuntu.com/8.04/serverguide/C/libvirt.html it states that the dhcdb daemon will need to be stopped and disabled if used (Desktop installs like mine).  It says to do the following:
    • sudo /etc/init.d/dhcdbd stop
      • this supposedly shuts down the service, but the command was not found on my computer and indeed the file /etc/init.d/dhcdbd is non-existent so apparently my version of Linux Mint uses something else.
  • Alternate configuration of /etc/network/interfaces was found at the bottom of this page: https://help.ubuntu.com/community/BridgingNetworkInterfaces
    • /etc/network/interfaces to read:
      • auto eth1
      • iface eth1 inet manual
      • up ip link set eth1 up
      •  
      • auto br0
      • iface br0 inet manual
        • bridge_ports eth1
        • bridge_fd 0
        • bridge_hello 2
        • bridge_maxage 12
        • bridge_stp off
    • I have not tried this yet, but suppose it will work also, since I currently have not problems, no need to try here.  The point of my eth0 having no separate IP address listed is normal as eth0 traffic routes through br0.
  • THE COMMAND brctl gives access to control and view bridge functions.  Typing brctl directly gives command options and man brctl give more help :)
  • more on bridging: http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge#Does_DHCP_work_over.2Fthrough_a_bridge.3F
  • For advanced bridging info and vlan bridging: http://blog.loftninjas.org/2008/08/11/enterprise-networking-with-kvm-and-libvirt/
    • discusses configuring libvert networking in the following files
      • /etc/libvirt/qemu/networks/default.xml
        • opening this file on my computer shows the network configuration of my virtual bridge virbr0, which I have not yet used for a virtual machine as I am mostly using the standard bridged network
      • /etc/libvirt/qemu/domain.xml
        • on my computer, this is blank
      • Discusses how to define vlan bridges
      • Shows examples of configuration of domain.xml and /etc/newtork/interfaces files for  creating subinterfaces
    • points to libvirt networking documentation: http://libvirt.org/formatnetwork.html


2) Prepare Machine for receiving ConVirt 2.0
Decided to create a VM on my desktop for this.  Its creation is straight forward using Virtual Machine Manager.  I created a VM of Linux Mint 11 64bit Gnome, which is akin to my desktop setup.

OK, that did not work, so need to install an earlier version of Ubuntu as Linux mint 11 is based on 11.04 and the script only accounts for versions up to ubuntu 10.10, so maybe I will install an Ubuntu server as a VM on my desktop.... done... selecting only openSSH server for install.....
hmmmm, after trying the step by step convirt install, it failed, maybe they wanted ubuntu desktop instead of server....

Could try with an Ubuntu 10.04 desktop, then use the partner directory..... perhaps should have tried this first!

In the process of doing the above, I discovered that Virtual Machine Manager can connect to Virtual Machines on my server directly.... though right now it seems to have trouble connecting to local storage on the server.... likely some configuration changes needed.  If ConVirt 2.0 seems to be too unstable or difficult to use, maybe vanilla Virtual Machine Manager will do the trick... Guess I don't really have a need for the fancy stuff like live migration anyway...


3) Install ConVirt 2.0 CMS software 
on the VM inside my Linux Mint machine, I will follow the instructions at: http://www.convirture.com/wiki/index.php?title=C2_ubuntu_installation
I may later install the CMS directly on my desktop, but don't want to risk messing up my desktop configuration untill I have a bit more experience with ConVirt.

  • did wget for 3 files and untarred the first
  • ran first command of install_dependencies and ran into an error with libc6-xen not being installable.... may either have to manually go through the setup or use another virutual machine of an earlier distribution.
So, starting again on the Ubuntu Server 10.10 x64 VM created above, with only OpenSSH server installed.
  • sudo apt-get install wget sudo
    • this updated the sudo package
  • wget the 3 files
  • tar -xzf convirt-install-2.0.1.tar.gz
    • untars in the home directory
  • sudo ./convirt-install/install/cms/scripts/install_dependencies
    • installs the dependencies
    • entered "convirt" as the mysql root crediential, will change later
  • Setup innodb buffer and memory pool
    • found the my.cnf configturation file in different location than specified in instructions.  Was found in directory /etc/mysql
    • this file states at the top that golbal options are setup in file and user-specific are setup in ~/.my.cnf ... since this machine is only for CMS, I will set it up globally by adding the two lines to  /etc/mysql/my.cnf  in the mysqld section
      • sudo nano /etc/mysql/my.cnf
        • innodb_buffer_pool_size=1G
        • innodb_additional_mem_pool_size=20M
    • sudo /etc/init.d/mysql restart
      • restarts mysql... convirt instructions had typos here
    • untar the CMS tarball... done
    • TurboGears setup gives some errors:
      • EnvironmentError: mysql_config not found
      • Error: installing mysql-python
      • Error: Failed creating Turbogears2 environment
    • Whatever we do now probably will not work without fixing these errors,however
    • setup sql database
    • Run setup convirt ... which throws more errors about dependencies.... maybe they meant to install this on Ubuntu desktop edition, rather than server edition.
    • GRRRRR, getting frustrated

Maybe I will just install the appliance disk.... Now downloading on KVM server
  • Downloaded the server, then unpacked it, but could not start it using instructions on Convirt website, rather completed remote control setup of managed server using Virtual Machine Manager.  
    • tar -jxf convirt-appliance-2.0.1.tbz2
      • unpacked file
    • then created directory /mnt/sotrage/vm_disks
    • moved disk there (c2_appliance.disk.xm)
  • apt-get install chkconfig
    • installed chkconfig, but it did no good as the point was to turn on libvirtd, but it was not installed , however another service must be in its place
  • http://virt-manager.org/page/RemoteSSH used this guide to setup remote management over SSH, but did not need to turn on the service libvirtd (must be by another name on ubuntu server)  Could have used other VMM remote management protocols **** http://virt-manager.org/page/Main_Page
  • opened the machine via Virtual Machine Manager
    • define new storage pool for server connecting directory /mnt/storage/vm_disks to storage pool called vm_disks
    • created new virtual machine by importing existing disk image and defining the server, then starting
    • login as cms with password convirt
    • login with new credentials
    • start convirt
      • cd ~/convirt
      • ./convirt-ctl start
    • Start web browers on another computer and point to
      • http://[ip address of CMS]:8081
      • default credentials is admin:admin
      • change default credentials
3.5) Setup Convirt CMS to connect to managed servers
  • Start web browers on another computer and point to
    • http://[ip address of CMS]:8081
    • default credentials is admin:admin
    • change default credentials 
      • admin
      • my new password
  • Connecting
    • Created new server Pool by right clicking the data center
    • added server by right clicking the pool and adding, including credentials and connection info
    • installed VNC viewer and keys from CMS to server plus opened VNC ports
    • Things are running well, and I can install new machines and configure storage pools, etc.
    • However, I am finding that virt-manager VMM native to linux is now just as powerful and combine it with virsh shell, more so.  
    • ConVirt's formatting and arrangement seems a little disorganized and dated.  For example, settings for new virtual machines are not as well presented as in virt-manager and certainly not close to that of VirtualBox or 


4) Connecting Virtual Machine Manager (virt-manager) on desktop computer to Server
  • https://help.ubuntu.com/community/KVM/Networking?highlight=%28%5CbCategoryVirtualization%5Cb%29
    • installed libcap2-bin package
    • give qemu the inheritable CAP_NET_ADMIN capability as described
    • edited /usr/bin/qemu as suggested
  • Able to now define storage devices, create new virtual machines, etc.
  • Bridge networking works well on the server.

  • Running Virt-manger on server from another computer:  Curiosity based test.... if I was on another (linux) computer and did not have virt-manager installed, could I still manage virtual machines on the server?
    • ssh user@serverip  =to log into server from console on laptop
    • sudo apt-get install virt-manager =to install VMM gui
    • exit =to get out of ssh tunnel
    • ssh -X user@serverip  =to get back in with X shell enabled
    • vir-manager =to open vir-manager..... works well and I can connect to the server and see VMs and settings I applied remotely from virt-manager on my desktop computer.


Conclusion:

I will abandon ConVirt 2.0 and utilize Virt-manager.  

Pro's of virt-manager over ConVirt 2.0
  • Cleaner interface
  • Lower footprint and easier setup
  • Does everything a small shop needs
  • More options on disk management
  • No need to have a VM running constantly just for running the interface
  • Much more active development
  • Simple server configuration with minimal extra packets and processes running at host level
Cons of virt-manager
  • Need to run from within linux (or Mac via ssh -X with virt-manager installed on server)
    • if you have Linux or Mac anyway, not a big deal - or you could create a VM dedicated to running virt-manager, which is in effect what ConVirt required anyway.
    • you could set it up to tunnel in remotely and run virt-manager via remote x session
  • no gui network setup tools, need to be done in config files or via commands --- for now!
    • simple bridging or NATing is easy though
Pro's of ConVirt 2.0
  • Web interface - log in from anywhere
  • Better statistics
  • live-migration - not any more as virt-manager does this handily now
Cons of ConVirt 2.0
  • No longer seems to be focus on Convirture... now Enterprise Cloud ... will product fade away?
  • Complicated setup
  • Disorganized GUI
  • Few extra real features for the fuss
  • Requires root login to all managed servers