Now came the process of migrating the data over from my file server to the new array.  I didn’t want to fiddle with NFS or anything of that nature since I was booting off a Jaunty live CD.  SSH was certainly possible but then there was the overhead.  Rsync was another solution.  But then I recalled a extremely fast way with tar & netcat.  Combine that with pipeviewer and you’ll get a fast transfer complete with verbose information.

It’s very simple.  With two computers, source and dest, you simply run the following:

On source (with IP 192.168.0.10):

tar -cf - /path | pv | nc -l -p 8888 -q 5

This will tar up /path and pipe it to netcat, which is listening on port 8888 for an incoming connection.

On dest:

nc 192.168.0.10 | pv | tar -xf -

Connect to source and pipe the output to tar for extraction.

Credit to Peteris’ wonderful blog entry on pv.

On a related note, I’ve considered tinkering with creating my own local repository mirror.  Not that I have nearly enough machines to make it necessary, but it would be an entertaining exercise.  Even found a basic HOW-TO or two.  However I have heard of potential issues: it can take weeks to fully mirror several distributitions (several GB each) and with an incomplete repository it would be somewhat pointless to use.  Luckily there seems to be an easy solution with mod_rewrite.

Apticron is very easy to set up, it’s in the repositories and requires basically no configuration.  It will drop a script in /etc/cron.daily and that is about it, emailing any reports to root.  Of course this can be modified through a .forward or an entry in /etc/aliases.

Logcheck is fairly simple to set up as well – it is also in the repositories.  Once installed, edit the /etc/logcheck/logcheck.conf file to configure.  The first thing you will want to set is the REPORTLEVEL setting, options are “workstation”, “server” (default value), or “paranoid”.  I use server on mine, which gives a good amount of detail. I would advise against using paranoid unless the server is extremely locked down and users do not typically login.  Workstation is good for a desktop environment.  The only other variable I edited was SENDMAILTO.  Logcheck works by basically comparing each  logentry against a set of regular expressions and generate a report if it does not match.  I had to modify one or two regex’s slightly to fix false positives, if you want my changes just ask and I’ll send them over.

One other small gem I want to mention : gkrellm.  I use this on both my desktop and server, it is invaluable for providing real-time system performance metrics.  Sure, it does not have any logging capabilities and thus unsuitable in a large-scale environment but for keeping an eye on one or two boxes it fits the bill quite nicely.

Download or wiki.

• Firstly, now you can you use the same script across multiple boxes – the UUIDs are configurable per hostname
• A bug was fixed where the script would fail if the destination disks were not already mounted.
• You can also customize the rsync invocation on a host-basis as well.  This was needed on my desktop machine where a /home account was mounted via NFS on a different file system causing IO errors and subsequently skipping the file deletion.

One final enhancement I want to add is the ability to spin the backup disk down after rsync is complete – this will not only help to increase drive life but also help reduce power use (however small it may be).  For some odd reason on my machines whenever I stop (umount, sync, then spin-down) an internal disk it works for a few seconds, then the drive spins back up again and I’m seeing ATA link reset messages as if it was just being plugged in.  External drives connected with eSATA seem to work just fine however.  Need to look into that more.

Also shortly available:

• A similar but different script I use for syncing my RAID storage array with an backup external drive connected via eSATA.  It’s a bit of a hack in some spots, but the nice thing is that it is almost fully automated.  Use this on a machine with one of those eSATA docks and you have a good way of making a quick backup of an array or disk.
• Sample service account script implementation with the ‘chattr’ command.

Download or wiki.

1. Run ssh-keygen -t rsa.  I specified a simple passphrase for general-purpose logins.  We will be adding the second phrase-less key later.

   (adechiaro@desktop:pts/6)-(4/0 @ 17k)-(09:22 PM)
   ~$ ssh-keygen -t rsa
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/adechiaro/.ssh/id_rsa):
   Enter passphrase (empty for no passphrase):
   Enter same passphrase again:
   Your identification has been saved in id_rsa.
   Your public key has been saved in id_rsa.pub.
   The key fingerprint is:
   5d:34:f1:de:ad:be:ef:65:34:36:a4:0d:75:d6:3c:47 adechiaro@desktop
   The key's randomart image is:
   +--[ RSA 2048]----+
   ...
   +-----------------+
   (adechiaro@desktop:pts/6)-(4/0 @ 17k)-(9:22 PM)
   ~$ cat .ssh/id_rsa.pub
   ssh-rsa <key> adechiaro@desktop

2. Any machine you want to be able to connect to with this key, login and copy the contents of your public key (id_rsa.pub) to the authorized_keys file.  These are all in your $HOME/.ssh/ directory.  There are various ways to do this: you could copy the file over with scp and cat/append it, you could remote in to the host and cut & paste the data, if you had a large infrastructure you could use ssh-copy-id or similar custom script.  It’s up to you, something like what is below should work in the general case.  Also the <key> is your public key in base64 encoded format.

   desktop:~$ scp .ssh/id_rsa.pub adechiaro@server:~
   id_service.pub                           100%  399     0.4KB/s   00:00
   desktop:~$ ssh adechiaro@server
   server:~$ cat id_rsa.pub >> ~/.ssh/authorized_keys

3. Now for an example of making the key more secure, you can add additional options to the authorized_keys file.  These come before the “ssh-rsa <key>” part of the entry (prefix the line with them):
   1. from=”host1,host2,10.0.0.1″ – This will prevent the key from authenticating except the hosts listed here, canonical name or IP.
   2. command=”/usr/local/bin/myscript.sh”, no-pty – Executes command upon login.  You might want to combine this with no-pty which prevents tty allocation (shell login) so you can securely execute a remote command and without granting direct login access (we will be doing this for our service account)
   3. There are a lot of more options – read the sshd manpage under the authorized_keys section for more.
4. Create the second public key saved with a different name.  Leave this with an empty passphrase as will be used for automation.

   ssh-keygen -f id_service

5. You will need to either write a simple script or use an existing one for linking with this key.  This will be run every time the account is logged into, regardless of what command may be passed on the SSH command line (the command line arguments will be read by the script but not necessarily executed).  Here is a simple generic one I threw together for general purpose use (download or view in wiki).   I do not make any promises this is completely bugfree/secure nor am I liable for any consequences!  If you use mine, you will want to configure the COMMANDS variable within the script.  These are the specific commands which the service account will be permitted to run.  A command specified on the SSH command line which does not match any entry in COMMANDS will not be run.  Also, make sure the file is chmod 500 after you are done configuring it, it’s basically a homebrew sudoers file and vital no other users can read the contents.
6. Copy your second key to the other machines as in step 2.  You will want to to prefix this new entry with ‘command=”/usr/local/bin/service.sh”, no-pty’, adjusting the path and script name as needed.  You may also want to specify ‘no-port-forwarding,no-X11-forwarding’ as additional security measures.
7. Done!

Now to run, simply connect via ssh.  It will default to your id_rsa key:

desktop:~$ ssh adechiaro@server
Enter passphrase for key '/home/adechiaro/.ssh/id_rsa':
server:~$

If you want to use your service account in a script, call it in the following manner:

desktop:~$ ssh adechiaro@server -i ~/.ssh/id_service /usr/bin/whoami
Executing command: "/usr/bin/whoami"
adechiaro
desktop:~$ ssh adechiaro@server -i ~/.ssh/id_service /bin/hostname
Executing command: "/bin/hostname"
server
desktop:~$

We need to specify the full path to the key in this case.  Replace whoami or hostname with whatever script/app you want to run and assuming you pre-configured it the service.sh script correctly, it should run just fine without prompting you for the passphrase.  Please feel free to leave any question and/or comments.

References:

http://www.securityfocus.com/infocus/1810
http://standalone-sysadmin.blogspot.com/2008/11/host-to-host-security-with-ssh-keys.html
http://standalone-sysadmin.blogspot.com/2008/11/wacky-ssh-authorized-keys-tricks.html
http://it.toolbox.com/blogs/unix-sysadmin/playing-with-openssh-public-keys-28377
http://blog.funnelfiasco.com/?p=44

According to a few different sources there are a few of the attributes which are important to watch, although opinions can vary slightly.  You can also get descriptions of each attribute here.  Also keep in mind I’m taking RAW_VALUE here, this Linux Journal article helps explain the difference between VALUE/WORST/THRESH/RAW values, however the data can be a little difficult to interpret (I usually stick with RAW).  Basically the best idea here is to keep current backups (a whole book can be written about those alone) and to schedule regular SMART self-tests for all your drives.  I used the following configuration in /etc/smartd.conf:

/dev/sda -a -o on -S on -s (S/../.././12|L/../../3/1) -m root
/dev/sdb -a -o on -S on -s (S/../.././12|L/../../3/3) -m root

This will configure regular ’short’ scans at noon daily, and ‘long’ scans at 1 and 3 pm on Wednesdays.  Other parameters provide more verbose and frequent information (-a) and enabling autosave (-S) and automatic offline (-o).  -m simply tells to mail any warnings (if found) to root.  -M will do the same but send all reports instead of just warnings.

Update (Dec 05): Note – this is important.  Depending upon your distribution, you may need to modify your /etc/default/smartmontools file.  On my Intrepid boxes I needed to uncomment the line “start_smartd=yes”.  Without it, smartd would not start up automatically and furthermore any attempts to start it would silently fall back to the prompt without any error message.  I was wondering why my automated selftests were not being run…