Anthony’s Blog

I couldn’t help myself anymore.  Newegg had 1 TB green drives on sale…  again, and my old ~1 TB storage array was nearly full.  Since my random computer parts are scattered about, I grabbed the only extra working machine I could find and threw the 4 drives into it.  Unfortunately for me the machine was an old Athlon 64 3200+ so the RAID-5 syncing took some time (10+ hours, it was maxing out CPU at ~25MB/s).

Now came the process of migrating the data over from my file server to the new array.  I didn’t want to fiddle with NFS or anything of that nature since I was booting off a Jaunty live CD.  SSH was certainly possible but then there was the overhead.  Rsync was another solution.  But then I recalled a extremely fast way with tar & netcat.  Combine that with pipeviewer and you’ll get a fast transfer complete with verbose information.

It’s very simple.  With two computers, source and dest, you simply run the following:

On source (with IP 192.168.0.10):

tar -cf - /path | pv | nc -l -p 8888 -q 5

This will tar up /path and pipe it to netcat, which is listening on port 8888 for an incoming connection.

On dest:

nc 192.168.0.10 | pv | tar -xf -

Connect to source and pipe the output to tar for extraction.

Credit to Peteris’ wonderful blog entry on pv.

For those of you who are unaware, the latest Ubuntu release – Jaunty – was released several days ago.  Normally, the fastest way to get the latest version is to torrent an ISO…  the repositories are so overloaded attempting to do an upgrade is not even remotely possible.  However, there is an alternative I stumbled upon.  Instead of using the default Ubuntu repositories, select the fastest mirror, apt-get update, then upgrade away!  I was getting sustained rates of 300 KB/s without any issue during my upgrade.

On a related note, I’ve considered tinkering with creating my own local repository mirror.  Not that I have nearly enough machines to make it necessary, but it would be an entertaining exercise.  Even found a basic HOW-TO or two.  However I have heard of potential issues: it can take weeks to fully mirror several distributitions (several GB each) and with an incomplete repository it would be somewhat pointless to use.  Luckily there seems to be an easy solution with mod_rewrite.

I wanted to write a brief posting on some basic ways to help remotely administer Ubuntu/Debian boxes.  Over the past few months I’ve been tinkering with various methods of handling this and what I’ve come up with seems to work fairly well.  It basically consists of two applications: apticron, which monitors repositories for package updates, and logcheck, which monitors logs in for any security or other noteworthy entries.

Apticron is very easy to set up, it’s in the repositories and requires basically no configuration.  It will drop a script in /etc/cron.daily and that is about it, emailing any reports to root.  Of course this can be modified through a .forward or an entry in /etc/aliases.

Logcheck is fairly simple to set up as well – it is also in the repositories.  Once installed, edit the /etc/logcheck/logcheck.conf file to configure.  The first thing you will want to set is the REPORTLEVEL setting, options are “workstation”, “server” (default value), or “paranoid”.  I use server on mine, which gives a good amount of detail. I would advise against using paranoid unless the server is extremely locked down and users do not typically login.  Workstation is good for a desktop environment.  The only other variable I edited was SENDMAILTO.  Logcheck works by basically comparing each  logentry against a set of regular expressions and generate a report if it does not match.  I had to modify one or two regex’s slightly to fix false positives, if you want my changes just ask and I’ll send them over.

One other small gem I want to mention : gkrellm.  I use this on both my desktop and server, it is invaluable for providing real-time system performance metrics.  Sure, it does not have any logging capabilities and thus unsuitable in a large-scale environment but for keeping an eye on one or two boxes it fits the bill quite nicely.

Another great little tool I came across – iotop. It’s basically a top-like UI app that monitors the data from vmstat. Only installed it recently but looks promising.

Just wanted to note that I recently discovered a nice graphical tool for checking SMART drive health status and even run self-tests from.  Quite handy.  Read more about it here at Ubuntu Unleashed.

Simple question that was on my mind for some time – what is RedHat/Fedora’s chkconfig equivalent?  sysv-rc-conf is the answer.  Not installed by default, simple apt-get solves that.  Syntax is almost identical to what I remember.  However also keep in mind that runlevels are not necessarily the same across different distributions.  If my memory serves me correctly, runlevels 1 and 6 are the only that are guaranteed to be identical on all platforms, although 3 and 5 are usually multi-user mode without and with X respectively.  Hope this may help a few of you!

Finally got together and uploaded my second rsync script I mentioned earlier in this posting.  Instead of mirroring one internal drive (all partitions) to another, this will copy contents of one path to the external drive, optionally delete old files on removable disk from the previous rsync and finally eject/spin-down the drive if wanted.

Download or wiki.

I’ve been using my rsync mirror script for a few weeks now and have implemented an additional one or two tweaks after deploying it on my desktop system as well.

• Firstly, now you can you use the same script across multiple boxes – the UUIDs are configurable per hostname
• A bug was fixed where the script would fail if the destination disks were not already mounted.
• You can also customize the rsync invocation on a host-basis as well.  This was needed on my desktop machine where a /home account was mounted via NFS on a different file system causing IO errors and subsequently skipping the file deletion.

One final enhancement I want to add is the ability to spin the backup disk down after rsync is complete – this will not only help to increase drive life but also help reduce power use (however small it may be).  For some odd reason on my machines whenever I stop (umount, sync, then spin-down) an internal disk it works for a few seconds, then the drive spins back up again and I’m seeing ATA link reset messages as if it was just being plugged in.  External drives connected with eSATA seem to work just fine however.  Need to look into that more.

Also shortly available:

• A similar but different script I use for syncing my RAID storage array with an backup external drive connected via eSATA.  It’s a bit of a hack in some spots, but the nice thing is that it is almost fully automated.  Use this on a machine with one of those eSATA docks and you have a good way of making a quick backup of an array or disk.
• Sample service account script implementation with the ‘chattr’ command.

Download or wiki.

If you have read my brief posting Intrepid upgrade done I mentioned I would shorlty be implementing SSH keys for my systems.  This is a simple HOWTO to cover the steps I used.  In my case I’m implementing this only a small home network, please adjust as needed.  I will be setting up a key for my primary user account plus an additional phrase-less key used for automation purposes.  This second key will act as a service account, restricted to running only a few particular applications and/or scripts.

1. Run ssh-keygen -t rsa.  I specified a simple passphrase for general-purpose logins.  We will be adding the second phrase-less key later.

   (adechiaro@desktop:pts/6)-(4/0 @ 17k)-(09:22 PM)
   ~$ ssh-keygen -t rsa
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/adechiaro/.ssh/id_rsa):
   Enter passphrase (empty for no passphrase):
   Enter same passphrase again:
   Your identification has been saved in id_rsa.
   Your public key has been saved in id_rsa.pub.
   The key fingerprint is:
   5d:34:f1:de:ad:be:ef:65:34:36:a4:0d:75:d6:3c:47 adechiaro@desktop
   The key's randomart image is:
   +--[ RSA 2048]----+
   ...
   +-----------------+
   (adechiaro@desktop:pts/6)-(4/0 @ 17k)-(9:22 PM)
   ~$ cat .ssh/id_rsa.pub
   ssh-rsa <key> adechiaro@desktop

2. Any machine you want to be able to connect to with this key, login and copy the contents of your public key (id_rsa.pub) to the authorized_keys file.  These are all in your $HOME/.ssh/ directory.  There are various ways to do this: you could copy the file over with scp and cat/append it, you could remote in to the host and cut & paste the data, if you had a large infrastructure you could use ssh-copy-id or similar custom script.  It’s up to you, something like what is below should work in the general case.  Also the <key> is your public key in base64 encoded format.

   desktop:~$ scp .ssh/id_rsa.pub adechiaro@server:~
   id_service.pub                           100%  399     0.4KB/s   00:00
   desktop:~$ ssh adechiaro@server
   server:~$ cat id_rsa.pub >> ~/.ssh/authorized_keys

3. Now for an example of making the key more secure, you can add additional options to the authorized_keys file.  These come before the “ssh-rsa <key>” part of the entry (prefix the line with them): Read the rest of this entry »

So after installing Intrepid on my new new Acer Aspire One netbook, I was working with SMART to resolving the Load_Cycle_Count issue (hard drive killer bug).  Once that was done I figured it would make sense to check the SMART health status of the rest of the drives in my desktops and server.  Good thing I did too…  results varied to a few counts of UDMA CRC Error Count to having hundreds of thousands of Hardware ECC Recovered, Raw Read Error Rate, and Seek Error Rates.  The RAID drives in my server had a few million counts of Seek Error Rate as well.  However I did run self-tests and all came back passed so it is something I am going to keep a close eye on (GkrellM plugin perhaps?)

According to a few different sources there are a few of the attributes which are important to watch, although opinions can vary slightly.  You can also get descriptions of each attribute here.  Also keep in mind I’m taking RAW_VALUE here, this Linux Journal article helps explain the difference between VALUE/WORST/THRESH/RAW values, however the data can be a little difficult to interpret (I usually stick with RAW).  Basically the best idea here is to keep current backups (a whole book can be written about those alone) and to schedule regular SMART self-tests for all your drives.  I used the following configuration in /etc/smartd.conf:

/dev/sda -a -o on -S on -s (S/../.././12|L/../../3/1) -m root
/dev/sdb -a -o on -S on -s (S/../.././12|L/../../3/3) -m root

This will configure regular ’short’ scans at noon daily, and ‘long’ scans at 1 and 3 pm on Wednesdays.  Other parameters provide more verbose and frequent information (-a) and enabling autosave (-S) and automatic offline (-o).  -m simply tells to mail any warnings (if found) to root.  -M will do the same but send all reports instead of just warnings.

Update (Dec 05): Note – this is important.  Depending upon your distribution, you may need to modify your /etc/default/smartmontools file.  On my Intrepid boxes I needed to uncomment the line “start_smartd=yes”.  Without it, smartd would not start up automatically and furthermore any attempts to start it would silently fall back to the prompt without any error message.  I was wondering why my automated selftests were not being run…