Booting LiveCD’s with Cobbler

As the size of my network grows it becomes increasingly more convenient to be able to boot a live CD rescue environment easily.  Even though most of the hosts are virtual and can be booted directly with an ISO I think it’s even better to PXE boot them directly from Cobbler since it’s already set up.  The documentation for booting live CD’s on Cobbler’s wiki did not seem to work for the Ubuntu Rescue Remix, it’s possible that some modifications need to be made for Ubuntu-based images (as the functionality provided in the livecd-iso-to-pxeboot script seems to be based on RedHat-style distros) but instead I used nfsroot which worked without issue.

First, you’ll want to mount your Live CD via loopback and run a Cobbler import:

# mount ubuntu-rescue-remix-12.04.iso /mnt -o loop
# cobbler import --name=ubuntu-rescue-remix-12.04 --path=/mnt --breed=ubuntu

Next I configured NFS, my configuration is a bit unique as I serve NFS from my file/VM server and Cobbler is a VM which has it’s own application files mounted via NFS as well:

# grep cobbler /etc/exports
/srv/nfs/cobbler <cobblervmip>(rw,async,subtree_check,no_root_squash)
/srv/nfs/cobbler/webroot/cobbler <dhcp-class-c>/24(ro,sync,subtree_check,no_wdelay,insecure_locks,no_root_squash,insecure)

Create the distro as usual and set the kernel options to boot from NFS:

# cobbler distro add --name=ubuntu-rescue-remix-12.04 --kernel=/var/www/cobbler/ks_mirror/ubuntu-rescue-remix-12.04/casper/vmlinuz --initrd=/var/www/cobbler/ks_mirror/ubuntu-rescue-remix-12.04/casper/initrd.gz
# cobbler distro edit --name=ubuntu-rescue-remix-12.04 --kopts='nfsroot=<cobbler IP>:/var/www/cobbler/ks_mirror/ubuntu-rescue-remix-12.04 ip=dhcp netboot=nfs boot=casper

There was one minor issue which I encountered while attempting to mount the nfsroot, “short read: 24 < 28“.  The only thing a Google search turned up was a posting which goes into a bit more detail on the source of the problem.  Apparently if you are have  configured a hosts.allow/deny file it will be an issue because the kernel will use NFSv3 for an nfsroot.  I mistakenly assumed NFS was working fine when I was booted up since my system was configured to use version 4.  Also, the tcp_wrapper hosts.allow/deny files do not recognize CIDR notation (unlike /etc/exports), you’ll have to use “192.168.0.” to specify a /24.

You will also need to add a profile to actually boot the live CD but that is straightforward. Other then that it should work without issue!

Chroot’ed BIND and Apparmor

Recently I set up a server running BIND on my network to serve as at alternative to updating host files for my VM’s…  previously I accomplished this via Puppet and it worked OK but needed to be changed.  There were some minor additions to the how-to to get it working for me in the chroot, most of them were related to Apparmor profiles but some libraries were needed as well.

These notes are assuming you’re running under Ubuntu 12.04 LTS, earlier releases (or later ones) will have different requirements.  I’m also using /var/chroot/named as the path for the chroot.

Once you create the chroot environment, you’ll need OpenSSL libraries:

# mkdir -p /var/chroot/named/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines
# cp /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so /var/chroot/named/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines

Then the Apparmor profile must be updated to reflect the chroot and library.  I just created a new file under ‘local':

root@ns1:~# cat /etc/apparmor.d/local/usr.sbin.named 
# Site-specific additions and overrides for usr.sbin.named.
# For more details, please see /etc/apparmor.d/local/README.
#

# named chroot
/var/chroot/named/** r,
/var/chroot/named/etc/bind/** r,
/var/chroot/named/var/lib/bind/ rw,
/var/chroot/named/var/lib/bind/** rw,
/var/chroot/named/var/cache/bind/ rw,
/var/chroot/named/var/cache/bind/** rw,

/var/chroot/named/var/run/named/named.pid w,
/var/chroot/named/var/run/named/session.key w,

/var/chroot/named/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so rm,

Don’t forget to restart Apparmor for the changes to take effect!

ZeroMQ libraries & stability / radarwatch update

I’ve been hacking with ZeroMQ for some time now…  reading through the excellent guide and working on the preliminary steps to incorporate it into the new version (migrated to GitHub from Google Code) of my Doppler radar analysis application.  Multi-threading support was on the list of original criterion and considering many of the other analysis features I’d like to eventually add, full distributed processing would be the best design.  I’m not an expert hand at SMP application design and although the implementation needs here would be fairly trivial, choosing a good library to handle the nasty bits would make things vastly easier.

So I tinkered with several of their samples designs, implemented a few of the patterns in my code and tested things out.  Initial results with just a simple multithreaded service seemed work OK at first glance but eventually dropped data caused issues so I decided to try something with reliability such as the paranoid pirate pattern (robust reliable queuing).  Thinking back, I should have realised something was up when it exhibited that issue on both inproc IPC and tcp/localhost.  No matter what I was trying to do porting examples into my code nothing remained stable; either it crashed immediately or processed a few messages then died.  C and C++ code showed similar results.  Running the examples on their own worked fine.  The front-end/client-side Python application worked fine.  I gave it a fresh shot with another example – the Majordomo pattern (service-oriented reliable queuing) again showed similar symptoms of instability.  Commenting out huge sections of my code thinking it may of had something to do with a broken pointer or some other oversight on my part had little to no results.  Eventually I started comparing the GCC invocations and noticed there were some minor differences.  After 5 minutes of testing the weeks of tinkering finally yielded some fruit… Google confirmed my suspicions: linking ZeroMQ with -pg for gprof support crashes the library.

It’s amazing how quick it can be to find what you’re looking for when you know exactly what you want.  At least now I can get back to implementing 0MQ in my code.

Provisioning Ubuntu VM’s with Cobbler

I’ve been playing with Cobbler at home lately now that my server was upgraded to an quad core with the magic vmx flag and ran into an issue deploying Ubuntu VM’s with it.  The install itself and import of the distro is pretty straight-forward, Canonical has some documentation on the process.  Koan can be used for provisoning VM’s (as is mentioned in the docs), however I have some prior Cobbler experience with CentOS and would like to develop this further with Ubuntu.

The problem I encountered was the following: during the install process Ub would not detect the virtual disks and an error is thrown, “no root filesystem is defined”.  I’m using the default KVM virtio disk bus type here and apparently the debian-installer will not detect these with the default configuration.  If you launch a shell and check, /dev/vda exists and running fdisk on it seems to suggest all is good.  Also, running the install via a CD/ISO works just fine as well.  The problem lies with Cobbler.

Eventually I narrowed it down to the preseed file.  When you import the distro it’s mentioned in the Ubuntu Cobbler preseed docs that a default preseed file is generated.  What it doesn’t mention is that something in this configuration is not compatible with virtio disk types.  I haven’t narrowed it down, instead I just copied the Ubuntu profile to a new one and changed the kickstart/preseed to /var/www/cobbler/ks_mirror/[ubuntu-distro-name]/preseed/ubuntu-server-minimalvm.seed.  I’ve got some more work to do on the preseed, I’m not all too familiar with them yet but planning to change that.

mdadm: Device or resource busy error

I recently needed to rebuild a RAID1 array after a reboot for some odd reason and afterwards I was unable to assemble the array.  mdadm came back and reported “Device or resource busy” on one of the drives.  I couldn’t figure out what the issue was originally as it wasn’t mounted and no other processes were using the drive via lsof.  Eventually I tracked it down to a changed UUID – my fstab was trying to mount the old mdadm array and it locked the resource.  I checked it by doing ‘ls’ on /dev/disk/by-uuid/.  Updating fstab with a new UUID, rebooting to clean up things and reassembling the array solved it.  Just a useful item to keep in mind.

Debugging logcheck rules and security events

I’ve been a big fan of logcheck for monitoring my servers, when properly configured it works very well and is pretty flexible.   Unless you are using a centralized logging system such as Splunk most of us are guilty of not thoroughly checking our logs.  I like to use logcheck to perform a simple audit of what my systems are up to, it’s not perfect but certainly better then nothing.

My configuration has been tweaked a bit, adding some custom regex’s for ignoring a few common items.  I found a nice debugging tip on a old posting from the logcheck-devel mailing list which mentions using egrep to test new rules:

cat <logfile> | egrep -v -f /etc/logcheck/ignore.d.workstation/regex

This has saved me a lot of time and frustration when making the final tweaks to a regex.  However, recently I had some difficulties ignoring what seemed to be particularly stubborn security events.  From best I could tell, grep suggested that my expression was filtering properly, yet logcheck was still reporting on these events.  Finally reading through the README I discovered patterns cancelling security alarms must be places in violations.ignore.d, not ignore.d.workstation/server.  Something to be mindful of.

NFS root on SheevaPlug with Debian

I broke out my SheevaPlug the other day and decided to tinker with it a bit… recently I had some new projects ideas where it would be of use.  I also wanted to configure it with an NFS root to avoid flash wear issues and a way to easily back up the device.  This is an attempt at documenting the process in the hope it may help someone else with all the issues I ran into.  YMMV, damnum absque injuria, etc…

The plug had previously been sitting in my closet for some time, in fact it’s been so long that the Ubuntu Jaunty 9.04 install it came with was very outdated by this point.  Unfortunately the idea of upgrading to any more recent Ubuntu release will not work as they are compiled for newer ARM architectures; the CPU in the SheevaPlug is only an ARMv5 and 9.04 was the last version to support this core.  Ubuntu 9.10 was built for v6 and 10.04 and later target the Cortex ARMv7 core.

However we’re still in luck as Debian has an ARM port which supports Marvell’s Kirkwood platform.  And thanks to some wonderful work by Martin we have an easy to follow process for installing Squeeze on the SheevaPlug.  There’s a few caveats and issues to contend with and you will probably have to upgrade U-Boot (Marvell shipped their own variant which added support for plug computers but newer versions support it directly now).  This is all explained very well on Martin’s Debian installation page.  I recommend choosing the option for loading the installer via TFTP as you will need it later if you want to use NFS root.  I also initially installed Debian on an SD card, for some reason trying a USB drive did not work.

Once the installation is done, boot into your new Squeeze install and prepare an NFS mount on another box somewhere which will serve your rootfs.  I perused the DisklessUbuntuHowTo doc to get a basic idea of how it works, you’ll be doing something similar except the initrd update process will be different – this will be explained below.  Once your NFS share is created on the server side, mount it on the SheevaPlug and copy the files over (obviously replacing the IP’s as necessary):

mount -t nfs 192.168.1.2:/nfsroot /mnt
cp -ax /. /mnt/.
cp -ax /boot/. /mnt/boot/.

The Ubuntu HOW-TO also lists copying /dev but this will not be needed as it is mounted via udev.  On my install /boot was a separate partition so make sure you include that as well.  Edit a few files in the new rootfs on the server-side to reflect a few things – /nfsroot/etc/network/interfaces to assign a static IP and /nfsroot/etc/fstab to replace your rootfs device with /dev/nfs.

You’ll also want to copy over the uImage and uInitrd files from /nfsroot/boot over to your TFTP folder as they’ll be needed to boot the plug.

Now comes the fun part, creating a new initrd.  Unfortunately the one installed by Debian did not support a NFS root so we need to build a new one.  Do this on the server-side as the new initrd will be copied to the TFTP folder as well.  The steps are relatively straight-forward but I recommend you read through the details first to get an better understanding.  The steps I outline are taken from a post on editing initrd’s and another on Debian SheevaPlug installation.

The process consists of the following: copy the kirkwood initrd image, extract it, update the initramfs.conf, re-compress, then rebuild it adding a special header for u-Boot.

~$ mkdir tmp

~$ cd tmp

~/tmp$ cp /home/exports/sheeva_squeeze_nfs_root/boot/initrd.img-2.6.32-5-kirkwood ./initrd.img-2.6.32-5-kirkwood.gz

~/tmp$ gunzip initrd.img-2.6.32-5-kirkwood.gz 

~/tmp$ mkdir initrd

~/tmp$ cd initrd/

~/tmp/initrd$ cpio -id < ../initrd.img-2.6.32-5-kirkwood
25329 blocks

~/tmp/initrd$ ls
bin  conf  etc  init  lib  sbin  scripts

At this point you’ll have the initrd extracted.  Edit the conf/initramfs.conf file and search for BOOT=local, change it to BOOT=nfs.

~/tmp/initrd$ vim conf/initramfs.conf

~/tmp/initrd$ find . | cpio --create --format='newc' > ../initrd.img-2.6.32-5-kirkwood-nfs
25329 blocks

~/tmp/initrd$ cd ..

~/tmp$ ls
initrd  initrd.img-2.6.32-5-kirkwood  initrd.img-2.6.32-5-kirkwood-nfs

~/tmp$ gzip -c initrd.img-2.6.32-5-kirkwood-nfs > initrd.img-2.6.32-5-kirkwood-nfs.gz

~/tmp$ ls
initrd  initrd.img-2.6.32-5-kirkwood  initrd.img-2.6.32-5-kirkwood-nfs  initrd.img-2.6.32-5-kirkwood-nfs.gz

~/tmp$ mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n 'Debian ramdisk' -d initrd.img-2.6.32-5-kirkwood-nfs.gz initrd.img-2.6.32-5-kirkwood-nfs
Image Name:   Debian ramdisk
Created:      Thu Oct 20 20:43:31 2011
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    5473115 Bytes = 5344.84 kB = 5.22 MB
Load Address: 0x00000000
Entry Point:  0x00000000

~/tmp$ ls
initrd  initrd.img-2.6.32-5-kirkwood  initrd.img-2.6.32-5-kirkwood-nfs  initrd.img-2.6.32-5-kirkwood-nfs.gz

~/tmp$ cp initrd.img-2.6.32-5-kirkwood-nfs /var/lib/tftpboot/uImage-nfsroot

Now that you have the new initrd created you’ll need to update your uBoot environment variables.  This can be quite complex in-and-of itself.  The previous u-Boot environment variables I had did not work with this new setup, my attempts at abstracting away most of the complexity into many variables did not seem to work, there were many instances where variables referenced in commands and other variables were not getting set properly.  I basically re-worked it from the ground up using sample material at DENX’s (developers of u-Boot) website.  Their manual has a lot of information, you’ll probably want to reference the CommandLineParsing, UBootCmdGroupEnvironment, and LinuxNfsRoot pages primarily.  The working copy of my u-Boot environment is listed below:

addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1
addmisc=setenv bootargs ${bootargs}
addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate}
baudrate=115200
boot_mmc=setenv bootargs $(bootargs_console); run bootcmd_mmc; bootm ${kernel_addr_r} ${fdt_addr_r}
boot_nfs=tftpboot ${kernel_addr_r} ${bootfile}; tftpboot ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty; bootm ${kernel_addr_r} ${fdt_addr_r}
bootargs=root=/dev/nfs rw nfsroot=192.168.1.2:/home/exports/sheeva_squeeze_nfs_root ip=192.168.1.11:192.168.1.2:192.168.1.1:255.255.255.0:sheeva:eth0:off panic=1 console=ttyS0,115200
bootargs_console=console=ttyS0,115200 mtdparts=orion_nand:0x00100000@0x00000000(uBoot)ro,0x00400000@0x00100000(uImage),0x1fb00000@0x00500000(rootfs)
bootargs_nfs=$(bootargs_console) root=/dev/nfs rw nfsroot=$(serverip):$(rootpath) ip=$(ipaddr):$(serverip):$(gateway):$(netmask):sheeva:eth0:none
bootcmd=run boot_nfs
bootcmd_mmc=mmc init; ext2load mmc 0:1 0x00800000 /uImage; ext2load mmc 0:1 0x01100000 /uInitrd
bootcmd_nfs=tftpboot ${kernel_addr_r} ${bootfile}; tftpboot ${fdt_addr_r} ${fdt_file}
bootdelay=3
bootfile=uImage.sheeva.squeeze
console=console=ttyS0,115200 mtdparts=nand_mtd:0x00100000@0x00000000(uBoot)ro,0x00400000@0x00100000(uImage),0x1fb00000@0x00500000(rootfs)
ethact=egiga0
ethaddr=00:50:43:xx:xx:xx
fdt_addr=0x01100000
fdt_addr_r=0x01100000
fdt_file=uInitrd-nfsroot
fileaddr=800000
filesize=539CDD
gatewayip=192.168.1.1
hostname=sheeva
ipaddr=192.168.1.11
kernel_addr_r=0x00800000
netdev=eth0
netmask=255.255.255.0
nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath}
rootpath=/home/exports/sheeva_squeeze_nfs_root
serverip=192.168.1.2
stderr=serial
stdin=serial
stdout=serial
x_bootargs=console=ttyS0,115200 mtdparts=orion_nand:512k(uboot),4m@1m(kernel),507m@5m(rootfs) rw
x_bootargs_root=ubi.mtd=2 root=ubi0:rootfs rootfstype=ubifs
x_bootcmd_kernel=nand read 0x6400000 0x100000 0x400000
x_bootcmd_sata=ide reset;
x_bootcmd_usb=usb start;

By default it’s set to boot from NFS via the bootcmd=run boot_nfs line.  If you’re having issues getting NFS root to work, you may want to remove the panic=1 from the addip variable, this way it can drop you to an initrd prompt.  The boot_mmc variable is adapted from Martin’s example.  You may need to modify these slightly if your SheevaPlug is different, specifically the bootargs_console variable – mtdparts reflects the flash partition setup on my plug, update if yours differs (I think these are the factory default settings), /proc/mtd will have this information in it if you boot again from the Debian install.  Also I changed it to orion_nand, I believe it used to be orion_mtd initially.

That’s basically it.   Hope this is of use to someone!

Verizon/Samsung 4G MiFi and Ubuntu

I recently got a MiFi card for my on-call rotation with work and had some issues getting it to play nice with the Ubuntu install on my laptop.  It’s a Samsung SCH-LC11 and various iMac’s in the office connected to it just fine.  My laptop would connect, then almost immediately disconnect.  Pretty much unusable.  I connected to the office wifi just fine so I know that wasn’t an issue.

A quick search found a solution on the ubuntu forums.  Basically, you need to connect to the device (obviously using a Windows or Mac) and log into the web admin page (default of 192.168.1.1).  Check the wifi configuration security; the encryption protocol is probably set to WPA with TKIP.  You need to set it to WPA2 with AES.

 

Samba shares multple connections to a shared resource

I don’t normally post much about Windows here as I have a limited presence with it at home – only a VM for Lightroom – but I came across this counter-intuitive problem earlier in the week.  When building a new VM I discovered for some odd reason Windows would not allow me to map multiple shares to my Samba server under different paths with the same credentials.  I kept getting the error:

Multiple connections to a server or shared resource by the same user,
using more then one user name, are not allowed.

Now I was using the same user name on all of these and it worked perfectly fine on my old 2k3 VM and I changed nothing on the Samba side.  No attempts at removing the shares and recreating them or reboots (even 3 of them!) seemed to help.  Sigh…  A bit of Googling suggested deleting all the shares explicitly on the command line, then try recreating them.  This seemed to fix it!

net use * /delete

More sensor interfacing with the Bus Pirate

After interfacing the TMP102 temperature sensor recently with the Bus Pirate, the next step on my agenda was to integrate this with the Web Platform along with a HH10D humidity sensor.  Unfortunately I realised a bit late (entirely my fault rushing an order) that although the HH10D was an I2C sensor, the humidity was actually read through a frequency output pin; the I2C EEPROM was only for calibration data.  Bus Pirate to the rescue once again!

This, like the TMP102 is an 3.3V sensor and needs pullup resistors on the I2C bus to read the EEPROM.  I wasted spent a number of hours with this getting nothing but garbage data with the first sensor I tried (the calibration values were completely wrong), on the second try with a backup sensor I had much better luck, getting good results almost immediately.

We’ll be doing a similar setup to the TMP102: using I2C and turning on the power supplies but in this case the AUX pin should be set to HIGH-Z for input/reading.

HiZ> m
1. HiZ
2. 1-WIRE
3. UART
4. I2C
5. SPI
6. 2WIRE
7. 3WIRE
8. KEYB
9. LCD
x. exit(without change)

(1)> 4
Set speed:
 1. ~5KHz
 2. ~50KHz
 3. ~100KHz
 4. ~400KHz

(1)> 3
Ready
I2C> c
a/A/@ controls AUX pin
I2C> @
AUX INPUT/HI-Z, READ: 0
I2C> W
Power supplies ON
I2C> (1)
Searching I2C address space. Found devices at:
0xA2(0x51 W) 0xA3(0x51 R)

Good.  Now if you take a look at the datasheet for the M24C02 EEPROM to do a random address read you’ll need to send a start bit, device select (read only; R/W set to 0), address, then another start bit, device select again (but with R/W set to 1), then read commands.  The HH10D datasheet mentions there are 2 calibration factors we need to read: sensitivity and offset and each are 2 byte integers, making a total of 4 bytes.  These start at address EEPROM address 10.  Using the BP, we do the following:

I2C> [0xa2 0x0a [0xa3 r:4
I2C START BIT
WRITE: 0xA2 ACK
WRITE: 0x0A ACK
I2C START BIT
WRITE: 0xA3 ACK
READ: 0x01  ACK 0x8B  ACK 0x1E  ACK 0x32
I2C>

This is in big-endian format so the sensitivity value (at address 0x10) reads 0x018B and the offset is 0x1E32.  We’ll also do a frequency read, the first as baseline with my AC running nearby and the second breathing on the sensor:

I2C> f
AUX Frequency:  autorange 7,169 Hz
I2C> f
AUX Frequency:  autorange 6,790 Hz

The formula in the datasheet states the RH = (offset-freq) * sensitivity/4096.  Plugging these in we get 54% and 90.6% RH which seems to match the other humidity sensor in my room.  Success!

Next step was getting this working on my Web Platform.  I have some very bare-bones code working with the TMP102 already, just need to work on the HH10D a bit more and I’ll post my example up here.