Anthony’s Blog

I couldn’t help myself anymore.  Newegg had 1 TB green drives on sale…  again, and my old ~1 TB storage array was nearly full.  Since my random computer parts are scattered about, I grabbed the only extra working machine I could find and threw the 4 drives into it.  Unfortunately for me the machine was an old Athlon 64 3200+ so the RAID-5 syncing took some time (10+ hours, it was maxing out CPU at ~25MB/s).

Now came the process of migrating the data over from my file server to the new array.  I didn’t want to fiddle with NFS or anything of that nature since I was booting off a Jaunty live CD.  SSH was certainly possible but then there was the overhead.  Rsync was another solution.  But then I recalled a extremely fast way with tar & netcat.  Combine that with pipeviewer and you’ll get a fast transfer complete with verbose information.

It’s very simple.  With two computers, source and dest, you simply run the following:

On source (with IP 192.168.0.10):

tar -cf - /path | pv | nc -l -p 8888 -q 5

This will tar up /path and pipe it to netcat, which is listening on port 8888 for an incoming connection.

On dest:

nc 192.168.0.10 | pv | tar -xf -

Connect to source and pipe the output to tar for extraction.

Credit to Peteris’ wonderful blog entry on pv.

For those of you who are unaware, the latest Ubuntu release – Jaunty – was released several days ago.  Normally, the fastest way to get the latest version is to torrent an ISO…  the repositories are so overloaded attempting to do an upgrade is not even remotely possible.  However, there is an alternative I stumbled upon.  Instead of using the default Ubuntu repositories, select the fastest mirror, apt-get update, then upgrade away!  I was getting sustained rates of 300 KB/s without any issue during my upgrade.

On a related note, I’ve considered tinkering with creating my own local repository mirror.  Not that I have nearly enough machines to make it necessary, but it would be an entertaining exercise.  Even found a basic HOW-TO or two.  However I have heard of potential issues: it can take weeks to fully mirror several distributitions (several GB each) and with an incomplete repository it would be somewhat pointless to use.  Luckily there seems to be an easy solution with mod_rewrite.

I thought documenting my notes and what I’ve leared about the SheevaPlug so far may be useful so I’m putting together a basic HOW-TO.  The documentation from GlobalScale Technology is a great place to start.

Firstly, it will likely come pre-installed with Ubuntu Jaunty Beta, although possibly Gentoo.  Both can run on the plug in addition to FreeBSD.  Wiki explains more.  It also has a good entry on configuring the plug to boot from an SD card rather then the onboard NAND flash.

The forums are a very useful resource as well:

Configuring the USB serial console (Linux & Windows).  My Ubuntu desktop failed to create a /dev/ttyUSB1 on it’s own when plugged in, a simple ‘modprobe ftdi_sio product=0×9e8f vendor=0×9e88′ resolved that.

Give care when flashing partitions or images and make sure you don’t overwrite /dev/mtd0 or /dev/mtdblock0 (1st MB of NAND), it holds the uBoot boot image, the plug cannot boot without it.  Next partition is the Linux kernel image (~1-4mb), and finally a root filesystem image (remainder of the 512MB).

I somehow managed to brick mine at one point, after a fsck and reboot I got a “Verifying Checksum … Bad Data CRC” error and it would not boot.  Playing with it a bit, I managed to get it to boot by tftp’ing a kernel image and mount a root filesystem via NFS.  GST’s documentation and the wiki provide plenty of guidance.  I also had issues getting it to re-flash the kernel image to /dev/mtd1 because the kernel image provided by GST was slightly too big for the default size of mtd1.  Since then I’ve read they have released a new kernel image that fits, but at the time I simply increased the size of mtd1 (to 4mb rather then the default 1mb) by editing the mtdparts setting in uBoot.

It also occured to me that coming up with a makeshift bootmenu might be useful for debugging:

set console 'console=ttyS0,115200 mtdparts=nand_mtd:0x00100000@0x00000000(uBoot)ro,0x00400000@0x00100000(uImage),0x1fb00000@0x00500000(rootfs)'

set boottftp 'tftpboot 0x2000000 $(image_name)'
set bootargs_nfs 'root=/dev/nfs rw'
set boot_nfs 'run boottftp;setenv bootargs $(console) $(bootargs_nfs) nfsroot=$(serverip):$(rootpath) ip=$(ipaddr):$(serverip)$(bootargs_end);bootm 0x2000000'

set bootnand 'nand read.e 0x2000000 0x00100000 0x00500000'
set bootargs_nand 'root=/dev/mtdblock2 rw'
set boot_nand 'run bootnand;setenv bootargs $(console) $(bootargs_nand) ip=$(ipaddr):$(serverip)$(bootargs_end);bootm 0x2000000'

set bootargs_sd 'root=/dev/mmcblk0p1 rw'
set boot_sd 'run bootnand;setenv bootargs $(console) $(bootargs_sd) ip=$(ipaddr):$(serverip)$(bootargs_end);bootm 0x2000000'

set bootcmd 'run boot_sd;'

As I mentioned, the mtdparts I provided are configured for a 4mb mtd1 partition, you may need to modify this.  This thread explains mtdparts better.  All of the other variables such as IP, SERVERIP, etc need to be configured in addition to these listed here.  You will also see it’s set to boot from SD card by default, it can be changed to ‘run boot_nand’ or ‘run boot_nfs’ if you like.

So I’ve decided to disable my weekly Twitter updates after seeing how annoying it would be to read.  I’ve been working on a few things lately, first is a “DIY radar” weather warning system with the help of NOAA WSR-88D radar data, libGD, and Perl.  I recall some very basic image processing algorithms and code I tinkered with in BASIC when I was much younger, but it was very simplistic and this was ages ago.  It’s fun to catch up on some old programming interests.  I’ll throw a post about it together once the code is a bit more complete.

My Linux-based SheevaPlug (embedded Linux in a wall-wart) has been occupying some time as well, tinkering with the Jaunty install and NFS and SSH.  I’m pretty impressed with the capabilities of this so far, despite the fact I managed to partially brick it for awhile. I eventually want to build a Tweet-A-Watt and use my plug for interfacing.  Wiring my 1-wire weather station to the plug makes sense as well.

I wanted to write a brief posting on some basic ways to help remotely administer Ubuntu/Debian boxes.  Over the past few months I’ve been tinkering with various methods of handling this and what I’ve come up with seems to work fairly well.  It basically consists of two applications: apticron, which monitors repositories for package updates, and logcheck, which monitors logs in for any security or other noteworthy entries.

Apticron is very easy to set up, it’s in the repositories and requires basically no configuration.  It will drop a script in /etc/cron.daily and that is about it, emailing any reports to root.  Of course this can be modified through a .forward or an entry in /etc/aliases.

Logcheck is fairly simple to set up as well – it is also in the repositories.  Once installed, edit the /etc/logcheck/logcheck.conf file to configure.  The first thing you will want to set is the REPORTLEVEL setting, options are “workstation”, “server” (default value), or “paranoid”.  I use server on mine, which gives a good amount of detail. I would advise against using paranoid unless the server is extremely locked down and users do not typically login.  Workstation is good for a desktop environment.  The only other variable I edited was SENDMAILTO.  Logcheck works by basically comparing each  logentry against a set of regular expressions and generate a report if it does not match.  I had to modify one or two regex’s slightly to fix false positives, if you want my changes just ask and I’ll send them over.

One other small gem I want to mention : gkrellm.  I use this on both my desktop and server, it is invaluable for providing real-time system performance metrics.  Sure, it does not have any logging capabilities and thus unsuitable in a large-scale environment but for keeping an eye on one or two boxes it fits the bill quite nicely.

Another great little tool I came across – iotop. It’s basically a top-like UI app that monitors the data from vmstat. Only installed it recently but looks promising.

Just wanted to note that I recently discovered a nice graphical tool for checking SMART drive health status and even run self-tests from.  Quite handy.  Read more about it here at Ubuntu Unleashed.

Simple question that was on my mind for some time – what is RedHat/Fedora’s chkconfig equivalent?  sysv-rc-conf is the answer.  Not installed by default, simple apt-get solves that.  Syntax is almost identical to what I remember.  However also keep in mind that runlevels are not necessarily the same across different distributions.  If my memory serves me correctly, runlevels 1 and 6 are the only that are guaranteed to be identical on all platforms, although 3 and 5 are usually multi-user mode without and with X respectively.  Hope this may help a few of you!

So life has been quite hectic lately with the holidays and it’s time to get back to writing.  I still need to upload the final piece of my service account script which basically consists of a simple Nautilus script to invoke it.  This will be coming shortly, hopefully later this week.

As for my OS Ktux, I am uploading a tarfile with the latest code for anyone whom may care to peruse the source.  Unfortunately it’s been longer then I would have liked since this has been worked on.  However with my new laptop I will be re-creating my development environment and should have more information available soon.

• Ktux tarfile
• Development environment setup docs

Finally got together and uploaded my second rsync script I mentioned earlier in this posting.  Instead of mirroring one internal drive (all partitions) to another, this will copy contents of one path to the external drive, optionally delete old files on removable disk from the previous rsync and finally eject/spin-down the drive if wanted.

Download or wiki.