Booting LiveCD’s with Cobbler

As the size of my network grows it becomes increasingly more convenient to be able to boot a live CD rescue environment easily.  Even though most of the hosts are virtual and can be booted directly with an ISO I think it’s even better to PXE boot them directly from Cobbler since it’s already set up.  The documentation for booting live CD’s on Cobbler’s wiki did not seem to work for the Ubuntu Rescue Remix, it’s possible that some modifications need to be made for Ubuntu-based images (as the functionality provided in the livecd-iso-to-pxeboot script seems to be based on RedHat-style distros) but instead I used nfsroot which worked without issue.

First, you’ll want to mount your Live CD via loopback and run a Cobbler import:

# mount ubuntu-rescue-remix-12.04.iso /mnt -o loop
# cobbler import --name=ubuntu-rescue-remix-12.04 --path=/mnt --breed=ubuntu

Next I configured NFS, my configuration is a bit unique as I serve NFS from my file/VM server and Cobbler is a VM which has it’s own application files mounted via NFS as well:

# grep cobbler /etc/exports
/srv/nfs/cobbler <cobblervmip>(rw,async,subtree_check,no_root_squash)
/srv/nfs/cobbler/webroot/cobbler <dhcp-class-c>/24(ro,sync,subtree_check,no_wdelay,insecure_locks,no_root_squash,insecure)

Create the distro as usual and set the kernel options to boot from NFS:

# cobbler distro add --name=ubuntu-rescue-remix-12.04 --kernel=/var/www/cobbler/ks_mirror/ubuntu-rescue-remix-12.04/casper/vmlinuz --initrd=/var/www/cobbler/ks_mirror/ubuntu-rescue-remix-12.04/casper/initrd.gz
# cobbler distro edit --name=ubuntu-rescue-remix-12.04 --kopts='nfsroot=<cobbler IP>:/var/www/cobbler/ks_mirror/ubuntu-rescue-remix-12.04 ip=dhcp netboot=nfs boot=casper

There was one minor issue which I encountered while attempting to mount the nfsroot, “short read: 24 < 28“.  The only thing a Google search turned up was a posting which goes into a bit more detail on the source of the problem.  Apparently if you are have  configured a hosts.allow/deny file it will be an issue because the kernel will use NFSv3 for an nfsroot.  I mistakenly assumed NFS was working fine when I was booted up since my system was configured to use version 4.  Also, the tcp_wrapper hosts.allow/deny files do not recognize CIDR notation (unlike /etc/exports), you’ll have to use “192.168.0.” to specify a /24.

You will also need to add a profile to actually boot the live CD but that is straightforward. Other then that it should work without issue!

Chroot’ed BIND and Apparmor

Recently I set up a server running BIND on my network to serve as at alternative to updating host files for my VM’s…  previously I accomplished this via Puppet and it worked OK but needed to be changed.  There were some minor additions to the how-to to get it working for me in the chroot, most of them were related to Apparmor profiles but some libraries were needed as well.

These notes are assuming you’re running under Ubuntu 12.04 LTS, earlier releases (or later ones) will have different requirements.  I’m also using /var/chroot/named as the path for the chroot.

Once you create the chroot environment, you’ll need OpenSSL libraries:

# mkdir -p /var/chroot/named/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines
# cp /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/ /var/chroot/named/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines

Then the Apparmor profile must be updated to reflect the chroot and library.  I just created a new file under ‘local’:

root@ns1:~# cat /etc/apparmor.d/local/usr.sbin.named 
# Site-specific additions and overrides for usr.sbin.named.
# For more details, please see /etc/apparmor.d/local/README.

# named chroot
/var/chroot/named/** r,
/var/chroot/named/etc/bind/** r,
/var/chroot/named/var/lib/bind/ rw,
/var/chroot/named/var/lib/bind/** rw,
/var/chroot/named/var/cache/bind/ rw,
/var/chroot/named/var/cache/bind/** rw,

/var/chroot/named/var/run/named/ w,
/var/chroot/named/var/run/named/session.key w,

/var/chroot/named/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/ rm,

Don’t forget to restart Apparmor for the changes to take effect!

Provisioning Ubuntu VM’s with Cobbler

I’ve been playing with Cobbler at home lately now that my server was upgraded to an quad core with the magic vmx flag and ran into an issue deploying Ubuntu VM’s with it.  The install itself and import of the distro is pretty straight-forward, Canonical has some documentation on the process.  Koan can be used for provisoning VM’s (as is mentioned in the docs), however I have some prior Cobbler experience with CentOS and would like to develop this further with Ubuntu.

The problem I encountered was the following: during the install process Ub would not detect the virtual disks and an error is thrown, “no root filesystem is defined”.  I’m using the default KVM virtio disk bus type here and apparently the debian-installer will not detect these with the default configuration.  If you launch a shell and check, /dev/vda exists and running fdisk on it seems to suggest all is good.  Also, running the install via a CD/ISO works just fine as well.  The problem lies with Cobbler.

Eventually I narrowed it down to the preseed file.  When you import the distro it’s mentioned in the Ubuntu Cobbler preseed docs that a default preseed file is generated.  What it doesn’t mention is that something in this configuration is not compatible with virtio disk types.  I haven’t narrowed it down, instead I just copied the Ubuntu profile to a new one and changed the kickstart/preseed to /var/www/cobbler/ks_mirror/[ubuntu-distro-name]/preseed/ubuntu-server-minimalvm.seed.  I’ve got some more work to do on the preseed, I’m not all too familiar with them yet but planning to change that.

Verizon/Samsung 4G MiFi and Ubuntu

I recently got a MiFi card for my on-call rotation with work and had some issues getting it to play nice with the Ubuntu install on my laptop.  It’s a Samsung SCH-LC11 and various iMac’s in the office connected to it just fine.  My laptop would connect, then almost immediately disconnect.  Pretty much unusable.  I connected to the office wifi just fine so I know that wasn’t an issue.

A quick search found a solution on the ubuntu forums.  Basically, you need to connect to the device (obviously using a Windows or Mac) and log into the web admin page (default of  Check the wifi configuration security; the encryption protocol is probably set to WPA with TKIP.  You need to set it to WPA2 with AES.


Upgrading EOL Ubuntu installations

I have a number of Ubuntu boxes laying around and gotten a bit lazy keeping some of the lesser-used ones up to date.  I realized this after trying an apt-get update resulted in 404 errors, oops.  Since I couldn’t directly do a dist-upgrade I checked the Ubuntu wiki for upgrading EOL installations, the process is pretty simple.

All you basically need to do is update your /etc/apt/sources.list and replace (or whatever servers you are using) with, setting the release for your current distro correctly of course.  If it’s a desktop system you may need to install or ugprade update-manager package and/or ubuntu-desktop as well.  Then a simple aptitude update && aptitude safe-upgrade and do-release-upgrade should take care of your needs.  If you are multiple releases behind you will need to upgrade from one release to the next individually one at a time, you can’t skip directly to the latest so it may take some time.  Otherwise it’s pretty straightforward and from my experience thus far very pain-free which is always a plus.

Repository Management

For those of you who are unaware, the latest Ubuntu release – Jaunty – was released several days ago.  Normally, the fastest way to get the latest version is to torrent an ISO…  the repositories are so overloaded attempting to do an upgrade is not even remotely possible.  However, there is an alternative I stumbled upon.  Instead of using the default Ubuntu repositories, select the fastest mirror, apt-get update, then upgrade away!  I was getting sustained rates of 300 KB/s without any issue during my upgrade.

On a related note, I’ve considered tinkering with creating my own local repository mirror.  Not that I have nearly enough machines to make it necessary, but it would be an entertaining exercise.  Even found a basic HOW-TO or two.  However I have heard of potential issues: it can take weeks to fully mirror several distributitions (several GB each) and with an incomplete repository it would be somewhat pointless to use.  Luckily there seems to be an easy solution with mod_rewrite.

Ubuntu Server – First Intrepid boot FAIL

Well, the upgrade to Intrepid went smoothly during the install process itself.  However after a reboot, the system hung partially through boot and dropped to a initramfs shell claiming “cannot find root device /dev/disk/by-uuid/50128bb8…” and “Gave up waiting for root device.”  Wonderful.  Tinkered around a bit, tried mounting drives manually only they were not listed in /dev.  Attempted booting old 2.6.24-18 kernel which worked fine.  Aha, so it’s something related to new kernel.  Did a quick search which revealed the following bug:

Apparently on certain hardware the kernel has a bug which causes a long timeout for the SCSI/SATA bus.  It took a good 2-3 minutes on my system but when I left it idle while I was reading the bug report on my desktop system, a bunch more lines flew by from the initramfs prompt about ata bus reset and detecting new drives.  After that a simple ‘exit’ from prompt continued a normal boot.

It’s a fairly important bug but at least a workaround exists.  I’ve tinkered with adding the ‘rootdelay’ option to my menu.lst but have not found the best match yet.  Maybe I’ll just leave it as is, my server almost never gets rebooted.  You’re instilling me with a lot of confidence doc, I mean Intrepid.  Definitely going to make a full backup of my desktop machine before attempting upgrade on that one.

Upgrading Ubuntu boxes to Intrepid

So I am going through the process of upgrading my server to 8.10.  A quite useful HOWTO on can be found guiding through the process (they also document upgrading from Desktop version as well).

I was not sure which exact command to run given that my headless server obviously doesn’t have update-manager running.  The HOWTO covers usage of the ‘do-release-upgrade’ command.  Only thing I ran beforehand was my rootfs rsync script to make a backup copy of my OS drive incase the worst happen.

If this runs smoothly I will make a backup copy of my desktop rootfs drive and do a similar upgrade to Intrepid.  I am already aware of one or two things I’m not keen on with Intrepid, notably that btnx is not compatible!  For those not aware, btnx was the premier application for configuring and making use of every single one of those buttons on the higher-end mice.  I have a Logitech MX Laser something and have it set up perfectly, tilt wheel left/right for forward/back in Firefox, extra buttons for minimize or close windows (Ctrl-W), etc.  I spend weeks trying to get it working the way I wanted with xmodmap and that ended in nothing but frustration.  I’m sure there will be some other things that don’t work quite the way I would like so a mirrored backup drive pre-upgrade is nice to have.