NFS root on SheevaPlug with Debian

I broke out my SheevaPlug the other day and decided to tinker with it a bit… recently I had some new projects ideas where it would be of use.  I also wanted to configure it with an NFS root to avoid flash wear issues and a way to easily back up the device.  This is an attempt at documenting the process in the hope it may help someone else with all the issues I ran into.  YMMV, damnum absque injuria, etc…

The plug had previously been sitting in my closet for some time, in fact it’s been so long that the Ubuntu Jaunty 9.04 install it came with was very outdated by this point.  Unfortunately the idea of upgrading to any more recent Ubuntu release will not work as they are compiled for newer ARM architectures; the CPU in the SheevaPlug is only an ARMv5 and 9.04 was the last version to support this core.  Ubuntu 9.10 was built for v6 and 10.04 and later target the Cortex ARMv7 core.

However we’re still in luck as Debian has an ARM port which supports Marvell’s Kirkwood platform.  And thanks to some wonderful work by Martin we have an easy to follow process for installing Squeeze on the SheevaPlug.  There’s a few caveats and issues to contend with and you will probably have to upgrade U-Boot (Marvell shipped their own variant which added support for plug computers but newer versions support it directly now).  This is all explained very well on Martin’s Debian installation page.  I recommend choosing the option for loading the installer via TFTP as you will need it later if you want to use NFS root.  I also initially installed Debian on an SD card, for some reason trying a USB drive did not work.

Once the installation is done, boot into your new Squeeze install and prepare an NFS mount on another box somewhere which will serve your rootfs.  I perused the DisklessUbuntuHowTo doc to get a basic idea of how it works, you’ll be doing something similar except the initrd update process will be different – this will be explained below.  Once your NFS share is created on the server side, mount it on the SheevaPlug and copy the files over (obviously replacing the IP’s as necessary):

mount -t nfs 192.168.1.2:/nfsroot /mnt
cp -ax /. /mnt/.
cp -ax /boot/. /mnt/boot/.

The Ubuntu HOW-TO also lists copying /dev but this will not be needed as it is mounted via udev.  On my install /boot was a separate partition so make sure you include that as well.  Edit a few files in the new rootfs on the server-side to reflect a few things – /nfsroot/etc/network/interfaces to assign a static IP and /nfsroot/etc/fstab to replace your rootfs device with /dev/nfs.

You’ll also want to copy over the uImage and uInitrd files from /nfsroot/boot over to your TFTP folder as they’ll be needed to boot the plug.

Now comes the fun part, creating a new initrd.  Unfortunately the one installed by Debian did not support a NFS root so we need to build a new one.  Do this on the server-side as the new initrd will be copied to the TFTP folder as well.  The steps are relatively straight-forward but I recommend you read through the details first to get an better understanding.  The steps I outline are taken from a post on editing initrd’s and another on Debian SheevaPlug installation.

The process consists of the following: copy the kirkwood initrd image, extract it, update the initramfs.conf, re-compress, then rebuild it adding a special header for u-Boot.

~$ mkdir tmp

~$ cd tmp

~/tmp$ cp /home/exports/sheeva_squeeze_nfs_root/boot/initrd.img-2.6.32-5-kirkwood ./initrd.img-2.6.32-5-kirkwood.gz

~/tmp$ gunzip initrd.img-2.6.32-5-kirkwood.gz 

~/tmp$ mkdir initrd

~/tmp$ cd initrd/

~/tmp/initrd$ cpio -id < ../initrd.img-2.6.32-5-kirkwood
25329 blocks

~/tmp/initrd$ ls
bin  conf  etc  init  lib  sbin  scripts

At this point you’ll have the initrd extracted.  Edit the conf/initramfs.conf file and search for BOOT=local, change it to BOOT=nfs.

~/tmp/initrd$ vim conf/initramfs.conf

~/tmp/initrd$ find . | cpio --create --format='newc' > ../initrd.img-2.6.32-5-kirkwood-nfs
25329 blocks

~/tmp/initrd$ cd ..

~/tmp$ ls
initrd  initrd.img-2.6.32-5-kirkwood  initrd.img-2.6.32-5-kirkwood-nfs

~/tmp$ gzip -c initrd.img-2.6.32-5-kirkwood-nfs > initrd.img-2.6.32-5-kirkwood-nfs.gz

~/tmp$ ls
initrd  initrd.img-2.6.32-5-kirkwood  initrd.img-2.6.32-5-kirkwood-nfs  initrd.img-2.6.32-5-kirkwood-nfs.gz

~/tmp$ mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n 'Debian ramdisk' -d initrd.img-2.6.32-5-kirkwood-nfs.gz initrd.img-2.6.32-5-kirkwood-nfs
Image Name:   Debian ramdisk
Created:      Thu Oct 20 20:43:31 2011
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    5473115 Bytes = 5344.84 kB = 5.22 MB
Load Address: 0x00000000
Entry Point:  0x00000000

~/tmp$ ls
initrd  initrd.img-2.6.32-5-kirkwood  initrd.img-2.6.32-5-kirkwood-nfs  initrd.img-2.6.32-5-kirkwood-nfs.gz

~/tmp$ cp initrd.img-2.6.32-5-kirkwood-nfs /var/lib/tftpboot/uImage-nfsroot

Now that you have the new initrd created you’ll need to update your uBoot environment variables.  This can be quite complex in-and-of itself.  The previous u-Boot environment variables I had did not work with this new setup, my attempts at abstracting away most of the complexity into many variables did not seem to work, there were many instances where variables referenced in commands and other variables were not getting set properly.  I basically re-worked it from the ground up using sample material at DENX’s (developers of u-Boot) website.  Their manual has a lot of information, you’ll probably want to reference the CommandLineParsing, UBootCmdGroupEnvironment, and LinuxNfsRoot pages primarily.  The working copy of my u-Boot environment is listed below:

addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1
addmisc=setenv bootargs ${bootargs}
addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate}
baudrate=115200
boot_mmc=setenv bootargs $(bootargs_console); run bootcmd_mmc; bootm ${kernel_addr_r} ${fdt_addr_r}
boot_nfs=tftpboot ${kernel_addr_r} ${bootfile}; tftpboot ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty; bootm ${kernel_addr_r} ${fdt_addr_r}
bootargs=root=/dev/nfs rw nfsroot=192.168.1.2:/home/exports/sheeva_squeeze_nfs_root ip=192.168.1.11:192.168.1.2:192.168.1.1:255.255.255.0:sheeva:eth0:off panic=1 console=ttyS0,115200
bootargs_console=console=ttyS0,115200 mtdparts=orion_nand:0x00100000@0x00000000(uBoot)ro,0x00400000@0x00100000(uImage),0x1fb00000@0x00500000(rootfs)
bootargs_nfs=$(bootargs_console) root=/dev/nfs rw nfsroot=$(serverip):$(rootpath) ip=$(ipaddr):$(serverip):$(gateway):$(netmask):sheeva:eth0:none
bootcmd=run boot_nfs
bootcmd_mmc=mmc init; ext2load mmc 0:1 0x00800000 /uImage; ext2load mmc 0:1 0x01100000 /uInitrd
bootcmd_nfs=tftpboot ${kernel_addr_r} ${bootfile}; tftpboot ${fdt_addr_r} ${fdt_file}
bootdelay=3
bootfile=uImage.sheeva.squeeze
console=console=ttyS0,115200 mtdparts=nand_mtd:0x00100000@0x00000000(uBoot)ro,0x00400000@0x00100000(uImage),0x1fb00000@0x00500000(rootfs)
ethact=egiga0
ethaddr=00:50:43:xx:xx:xx
fdt_addr=0x01100000
fdt_addr_r=0x01100000
fdt_file=uInitrd-nfsroot
fileaddr=800000
filesize=539CDD
gatewayip=192.168.1.1
hostname=sheeva
ipaddr=192.168.1.11
kernel_addr_r=0x00800000
netdev=eth0
netmask=255.255.255.0
nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath}
rootpath=/home/exports/sheeva_squeeze_nfs_root
serverip=192.168.1.2
stderr=serial
stdin=serial
stdout=serial
x_bootargs=console=ttyS0,115200 mtdparts=orion_nand:512k(uboot),4m@1m(kernel),507m@5m(rootfs) rw
x_bootargs_root=ubi.mtd=2 root=ubi0:rootfs rootfstype=ubifs
x_bootcmd_kernel=nand read 0x6400000 0x100000 0x400000
x_bootcmd_sata=ide reset;
x_bootcmd_usb=usb start;

By default it’s set to boot from NFS via the bootcmd=run boot_nfs line.  If you’re having issues getting NFS root to work, you may want to remove the panic=1 from the addip variable, this way it can drop you to an initrd prompt.  The boot_mmc variable is adapted from Martin’s example.  You may need to modify these slightly if your SheevaPlug is different, specifically the bootargs_console variable – mtdparts reflects the flash partition setup on my plug, update if yours differs (I think these are the factory default settings), /proc/mtd will have this information in it if you boot again from the Debian install.  Also I changed it to orion_nand, I believe it used to be orion_mtd initially.

That’s basically it.   Hope this is of use to someone!

Upgrading EOL Ubuntu installations

I have a number of Ubuntu boxes laying around and gotten a bit lazy keeping some of the lesser-used ones up to date.  I realized this after trying an apt-get update resulted in 404 errors, oops.  Since I couldn’t directly do a dist-upgrade I checked the Ubuntu wiki for upgrading EOL installations, the process is pretty simple.

All you basically need to do is update your /etc/apt/sources.list and replace us.archives.ubuntu.com (or whatever servers you are using) with old-releases.ubuntu.com, setting the release for your current distro correctly of course.  If it’s a desktop system you may need to install or ugprade update-manager package and/or ubuntu-desktop as well.  Then a simple aptitude update && aptitude safe-upgrade and do-release-upgrade should take care of your needs.  If you are multiple releases behind you will need to upgrade from one release to the next individually one at a time, you can’t skip directly to the latest so it may take some time.  Otherwise it’s pretty straightforward and from my experience thus far very pain-free which is always a plus.

Repository Management

For those of you who are unaware, the latest Ubuntu release – Jaunty – was released several days ago.  Normally, the fastest way to get the latest version is to torrent an ISO…  the repositories are so overloaded attempting to do an upgrade is not even remotely possible.  However, there is an alternative I stumbled upon.  Instead of using the default Ubuntu repositories, select the fastest mirror, apt-get update, then upgrade away!  I was getting sustained rates of 300 KB/s without any issue during my upgrade.

On a related note, I’ve considered tinkering with creating my own local repository mirror.  Not that I have nearly enough machines to make it necessary, but it would be an entertaining exercise.  Even found a basic HOW-TO or two.  However I have heard of potential issues: it can take weeks to fully mirror several distributitions (several GB each) and with an incomplete repository it would be somewhat pointless to use.  Luckily there seems to be an easy solution with mod_rewrite.

Intrepid upgrade done!

Overall I am fairly impressed with Intrepid.  They seemed to have fixed a number of bugs I encountered (especially that annoying menu pulldown delay with dual screen configurations), and I’ve actually switched back to full compiz mode.  Most things work fairly smoothly and performance seems a bit better as well.  There is one strange bug I’ve found where there is a vertical black bar on the right-hand side of my 2nd monitor, which cuts off the gnome-panel at the top of my screen.  What doesn’t make much sense to me is that maximized windows do not extend over into the dark limbo, unlike the panel:

Odd desktop/resolution bug

Odd desktop/resolution bug

Strange.  Maybe it didn’t auto-detect my old 2005FPW right, pretty sure I had a manually configured xorg.conf just prior to the upgrade  I’ll have to look into it further.

Regardless, now it comes time to design/implement more items that have been held back too long.  Firstly will entail setting up a basic SSH key repository for my various machines and accounts.  There are a number of good articles about this topic.  What I find particularly useful are the ‘from’ and ‘command’ options within OpenSSH (referenced in the last two URL’s).  I have been wondering about a solution to this for some time; I wanted to create passphrase-less SSH keys used for automation & scripting but did not want full access to anyone who may have compromised the keys.  This solution seems like it will fit perfectly!  Already have a few sample ideas to use this with.

After setting up SSH keys, next will come full NIS implementation.  I already have it installed on my server, but not configured or running at the current time.  NFS-mounted home directories always seem like a good option, and I would normally agree but I am not sure if I want the usability of my desktop system tied dependent upon the server being up and running.  Right now if the server goes down, I can still function fine on my desktop – just without access to my RAID storage array.  Also due to come will be a post on performance logging & monitoring, I love my GkrellM but something with metrics storage would be nice.

Ubuntu Server – First Intrepid boot FAIL

Well, the upgrade to Intrepid went smoothly during the install process itself.  However after a reboot, the system hung partially through boot and dropped to a initramfs shell claiming “cannot find root device /dev/disk/by-uuid/50128bb8…” and “Gave up waiting for root device.”  Wonderful.  Tinkered around a bit, tried mounting drives manually only they were not listed in /dev.  Attempted booting old 2.6.24-18 kernel which worked fine.  Aha, so it’s something related to new kernel.  Did a quick search which revealed the following bug:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/290153

Apparently on certain hardware the kernel has a bug which causes a long timeout for the SCSI/SATA bus.  It took a good 2-3 minutes on my system but when I left it idle while I was reading the bug report on my desktop system, a bunch more lines flew by from the initramfs prompt about ata bus reset and detecting new drives.  After that a simple ‘exit’ from prompt continued a normal boot.

It’s a fairly important bug but at least a workaround exists.  I’ve tinkered with adding the ‘rootdelay’ option to my menu.lst but have not found the best match yet.  Maybe I’ll just leave it as is, my server almost never gets rebooted.  You’re instilling me with a lot of confidence doc, I mean Intrepid.  Definitely going to make a full backup of my desktop machine before attempting upgrade on that one.

Upgrading Ubuntu boxes to Intrepid

So I am going through the process of upgrading my server to 8.10.  A quite useful HOWTO on howtoforge.com can be found guiding through the process (they also document upgrading from Desktop version as well).

I was not sure which exact command to run given that my headless server obviously doesn’t have update-manager running.  The HOWTO covers usage of the ‘do-release-upgrade’ command.  Only thing I ran beforehand was my rootfs rsync script to make a backup copy of my OS drive incase the worst happen.

If this runs smoothly I will make a backup copy of my desktop rootfs drive and do a similar upgrade to Intrepid.  I am already aware of one or two things I’m not keen on with Intrepid, notably that btnx is not compatible!  For those not aware, btnx was the premier application for configuring and making use of every single one of those buttons on the higher-end mice.  I have a Logitech MX Laser something and have it set up perfectly, tilt wheel left/right for forward/back in Firefox, extra buttons for minimize or close windows (Ctrl-W), etc.  I spend weeks trying to get it working the way I wanted with xmodmap and that ended in nothing but frustration.  I’m sure there will be some other things that don’t work quite the way I would like so a mirrored backup drive pre-upgrade is nice to have.