Saturday, August 25, 2012

Linux Lab in the Lap

 I am a RHCE and have been working toward getting a RHCA which involves sitting five "certificate of expertise" exams which are deep dive technical exams on various Red Hat products such as satellite, cluster, naming services and a couple of others.  To help me prepare for the exams I wanted to set up a lab environment that emulated as closely as possible the one used in the training courses that are associated with the exams.  Fortunately I had been generous with most of the specifications on my laptop so, apart from disk space, I had the capability to set up a virtual environment on my laptop.  I upgraded the laptop harddisk to give me the room and developed the "lab in the lap".

Originally I developed this running on RHEL 5 which was reasonably challenging since it involved a bunch of manual tweaking and hacking on xen infrastructure scripts directly (even finding out what to do proved difficult).  I have recently updated the environment to RHEL 6 which
proved to be much easier to configure up.

This picture shows that the lab configuration looks like:

There are three isolated networks cluster, iscsi1 and iscsi2.  Another network, appnet, is NAT'ed to the hosts network interface to provide external access if required.  Each machine has four network interfaces.  All three VM nodes (node 1 - 3) are built by first creating a template node (call it node0) using kickstart.  The subsequent nodes are created by cloning node0.  To simplify the cloning process the IP addresses for all the clone nodes are assigned using DHCP - this means that there are
no node specific actions to be done.

To create the setup I first set up all the networks using the virtual machine manager.  This is a bit tedious to do but not difficult.  In RHEL 6 you can also use the VMM to assign addresses to the virbr NICs
that are created when you create an isolated network, this provides the host access to the isolated networks.  In RHEL 5 setting up the network interfaces was a bit trickier, first I had to make sure I had enough "dummy" network interfaces by adding:

options dummy numdummies=5

to /etc/modprobe.conf.  Then I created a script in /etc/xen/scripts called multi-network-bridge which set up the multiple bridges and the associated dummy interfaces:

/etc/xen/scripts/network-bridge $@ vifnum=0 netdev=eth0 bridge=xenbr0
/etc/xen/scripts/network-bridge $@ vifnum=1 netdev=dummy0 bridge=virbr1
ifconfig dummy0 netmask broadcast up
/etc/xen/scripts/network-bridge $@ vifnum=2 netdev=dummy1 bridge=virbr2
ifconfig dummy1 netmask broadcast up
/etc/xen/scripts/network-bridge $@ vifnum=3 netdev=dummy2 bridge=virbr3
ifconfig dummy2 netmask broadcast up
/etc/xen/scripts/network-bridge $@ vifnum=4 netdev=dummy3 bridge=virbr4
ifconfig dummy3 netmask broadcast up

Finally, I modified /etc/xen/xend-config.sxp and replaced argument for the network-script entry with my own multi-network-bridge script so that xend would run my script on start up.

For the isolated networks, iptables was blocking network access to the host.  Libvirtd automatically configures iptables to provide the NAT facility for guests.  I added a rule to allow the network access on any port to allow the guests to communicate with the host on the other ethernet interfaces.  Unfortunately you cannot just reload the iptables rules because this flushes the rules added by libvirtd which breaks the virtual networking.

With the networks set up it was then time to kickstart node0, the template to be used to stamp out the other nodes.  On my host I set up directory with the contents of the RHEL distro under /var/www/html/rhel-5.6 and pointed the yum host config at this.  I installed httpd to support a network kickstart.  I dropped a basic kickstart file into /var/www/html/ks and then used:

virt-install --name=node0 --ram=512 --vcpus=1 --disk path=/var/lib/images/node0.img,size=8 --extraargs="ksdevice=eth0 ks=" --mac 00:16:3e:00:00:00 --network network:cluster --network network:appnet --network network:iscsi1 --network network:iscsi2 --location=/var/www/html/rhel-5.6/

Where is the IP address of one of the dummy/virbr-nic interfaces on the host machine.  The MAC address comes from the manually allocated xen ethernet range.  To make things simple I allocated the MAC addresses with the second to last octet being the node number and the
last octet being the interface number of the VM.  This made the scripting of the cloning simpler.  With this scheme I configured static DHCP allocations for all the nodes.  This can be done using "virsh
net-edit net_name", e.g:

virsh net-edit cluster

to edit the cluster net and add the static definitions in the dhcp section of the xml list this:

  <bridge name='virbr2' stp='on' delay='0' />
  <ip address='' netmask=''>
      <range start='' end='' />
      <host mac='00:16:3e:00:01:00' ip='' name='node1' />
      <host mac='00:16:3e:00:02:00' ip='' name='node2' />
      <host mac='00:16:3e:00:03:00' ip='' name='node3' />

The host entries above are the static address allocations.  Once this file is edited I found I had to manually copy the xml file from /etc/libvirt/qemu/networks into /var/lib/libvirt/network/.  It seems that the VMM manages the files in both places but virsh doesn't and the changes will not take effect.  In RHEL6 I found I could restart libvirtd (service libvirtd restart), then send a SIGHUP to dnsmasq, then restart libvirtd once again.  The first restart writes out the dnsmasq control files, then we can kick dnsmasq, then restart libvirtd again because it seems to lose contact with dnsmasq following the SIGHUP which breaks DHCP.  I don't know if this process works on RHEL5, I just rebooted after making changes back then.

Once the template node was built I cloned it using:

virt-clone -o node0 --force --mac 00:16:3e:00:01:00 --mac 00:16:3e:00:01:03 --mac 00:16:3e:00:01:01 --mac 00:16:3e:00:01:02 --name node1 --file /var/lib/images/node1.img

Doing a clone is slightly faster than building the node using kickstart, once the node is cloned I set the VM to autostart on boot and start up:

virsh autostart node1
virsh start node1

That's it, a node ready for experimenting with.  The labs I was doing called for up to three nodes running, this was a simple matter of creating more clones.  I wrote a shell script that manages the building and cloning of the nodes, the command line arguments are similar to those of the Red Hat script used in the actual labs though I have added extensions to build more/different nodes so I can use the lab setup for more than one course.

I found when building the RHEL 6.2 clones that the ethernet interfaces were numbered eth4 to eth7 instead of eth0 to eth3.  This was due to a set of udev rules that are intended to attach a particular ethernet interface to a particular device.  This makes sense in a real world machine because it keeps the interface numbering consistent even when an intferace goes missing but, in my case, this feature was not desirable so I had to add a post action to the RHEL 6.2 kickstart to remove the udev rules.  I have also noticed that since upgrading the host to RHEL 6.2 that my cloned machines no longer boot properly when the template machine is not built with a graphical console.  This used to work fine
when the host OS was RHEL 5.6.  Trying to debug this is a bit awkward because there is no console access nor is the networking coming up.  I guess I could loop mount the disk image and see if there are any logs but for the moment I can put up with making the template node with a graphical install.

  1. For those 1337 h4x0rz out there interested in cracking the root password in the linked kickstart file, here's a hint... try redhat to save yourself a bit of time.
  2. The course I attended wasn't actually on RHEL6 it was run on RHEL5.8, not that this really presented an issue as I just added the distro files in and setup a kickstarted template for that OS version.  I was able to complete all the labs with this setup.  The practice really helped in the exam, I was able to complete the exam in half the allotted time with a score of 100%.

Friday, August 03, 2012

lab in the lap adventures

To help me with practicing for a Red Hat certification exam I set up RHEL 5.6 on my laptop.  I configured up xen and was using a bunch of virtual machines, each with multiple network interfaces connected to isolated networks.  For convenience all the interfaces were configured using DHCP, this way I could build one VM and just clone it to make the other machines - during the clone the network interfaces were given different mac addresses and DHCP just handed out the static allocations for those MAC addresses.  This all worked quite smoothly, though it did take a lot of work and searching to put all the bits together and it did help me a lot in going over all the lab exercises before my exam.

I am going to do another course and exam soon but the course will be using RHEL 6.2 for the OS so I thought it would be good to update my install.  Red Hat's recommendation is that you do a clean install between major versions but I really didn't want to lose my xen configuration so I attempted to perform an upgrade instead which you can do by hitting escape within the first 60 seconds of the cdrom boot and then adding:

linux upgradeany

to the boot command line.  This forces the upgrade regardless of the version.  The upgrade seemed to work ok but the first hurdle was the process had not written a new entry in grub.conf to boot the upgraded system.  This was easy to fix, just boot up the cd in rescue mode and fix.  I then tried to get X working properly but no matter what I did I would either get a black screen or X would come up and just hang.  In the end I decided that it probably was going to take me more time to figure out what was wrong with X than it would to do a clean install and redo my VM setups.  After a clean install X worked fine, I had some problems initially with the synaptics touchpad but that was just a matter of configuring it in xorg.conf.

I had a USB stick with what I thought was a copy of the old /etc directory plus my VM build scripts and the like.  On closer inspection I found these files were a bit out of date.  I did do a dd of the entire linux disk to a file on a NetBSD machine so I knew I could get the files back... the trick was how.  I could have just put the image back onto my laptop disk but that would have been tedious, having to copy the data over, boot up the old OS, get what I wanted and then reinstall 6.2 again.  Fortunately, the NetBSD machine with the dd image on it had enough room to have rhel image uncompressed.  I though that what I could do is us vnconfig to create a virtual drive from the file like this:

vnconfig -c vnd0 ./dd_image_file

and then use lvm on NetBSD to access the logical volumes.  Unfortunately it looks like the version of the NetBSD kernel I have on that machine is too old and it had issues with lvm.  I knew that NetBSD on my laptop worked fine with lvm so what I ended up doing was NFS mounting the file system from the machine with the image (I didn't have enough room on the laptop to hold the disk image locally), then use vnconfig as above to create the virtual disk.  I then did:

lvm vgscan

which found the lvm volgroups on vnd0, I then did:

lvm vgchange -a y
mount -t ext2fs /dev/mapper/volgroup-lvol00 /mnt

an there we have it.  A disk image on one machine, nfs mounted to another, through a virtual disk driver into lvm.  It is a bit tortuous (and a bit slow due to the NFS mount going over wireless) but I can access the files and pull off what I want without having to reinstall.

When I get organised I will do a write up about the lab in the lap.