Anthony’s Blog

I wanted to write a brief posting on some basic ways to help remotely administer Ubuntu/Debian boxes.  Over the past few months I’ve been tinkering with various methods of handling this and what I’ve come up with seems to work fairly well.  It basically consists of two applications: apticron, which monitors repositories for package updates, and logcheck, which monitors logs in for any security or other noteworthy entries.

Apticron is very easy to set up, it’s in the repositories and requires basically no configuration.  It will drop a script in /etc/cron.daily and that is about it, emailing any reports to root.  Of course this can be modified through a .forward or an entry in /etc/aliases.

Logcheck is fairly simple to set up as well – it is also in the repositories.  Once installed, edit the /etc/logcheck/logcheck.conf file to configure.  The first thing you will want to set is the REPORTLEVEL setting, options are “workstation”, “server” (default value), or “paranoid”.  I use server on mine, which gives a good amount of detail. I would advise against using paranoid unless the server is extremely locked down and users do not typically login.  Workstation is good for a desktop environment.  The only other variable I edited was SENDMAILTO.  Logcheck works by basically comparing each  logentry against a set of regular expressions and generate a report if it does not match.  I had to modify one or two regex’s slightly to fix false positives, if you want my changes just ask and I’ll send them over.

One other small gem I want to mention : gkrellm.  I use this on both my desktop and server, it is invaluable for providing real-time system performance metrics.  Sure, it does not have any logging capabilities and thus unsuitable in a large-scale environment but for keeping an eye on one or two boxes it fits the bill quite nicely.

I’ve been using my rsync mirror script for a few weeks now and have implemented an additional one or two tweaks after deploying it on my desktop system as well.

• Firstly, now you can you use the same script across multiple boxes – the UUIDs are configurable per hostname
• A bug was fixed where the script would fail if the destination disks were not already mounted.
• You can also customize the rsync invocation on a host-basis as well.  This was needed on my desktop machine where a /home account was mounted via NFS on a different file system causing IO errors and subsequently skipping the file deletion.

One final enhancement I want to add is the ability to spin the backup disk down after rsync is complete – this will not only help to increase drive life but also help reduce power use (however small it may be).  For some odd reason on my machines whenever I stop (umount, sync, then spin-down) an internal disk it works for a few seconds, then the drive spins back up again and I’m seeing ATA link reset messages as if it was just being plugged in.  External drives connected with eSATA seem to work just fine however.  Need to look into that more.

Also shortly available:

• A similar but different script I use for syncing my RAID storage array with an backup external drive connected via eSATA.  It’s a bit of a hack in some spots, but the nice thing is that it is almost fully automated.  Use this on a machine with one of those eSATA docks and you have a good way of making a quick backup of an array or disk.
• Sample service account script implementation with the ‘chattr’ command.

Download or wiki.

If you have read my brief posting Intrepid upgrade done I mentioned I would shorlty be implementing SSH keys for my systems.  This is a simple HOWTO to cover the steps I used.  In my case I’m implementing this only a small home network, please adjust as needed.  I will be setting up a key for my primary user account plus an additional phrase-less key used for automation purposes.  This second key will act as a service account, restricted to running only a few particular applications and/or scripts.

1. Run ssh-keygen -t rsa.  I specified a simple passphrase for general-purpose logins.  We will be adding the second phrase-less key later.

   (adechiaro@desktop:pts/6)-(4/0 @ 17k)-(09:22 PM)
   ~$ ssh-keygen -t rsa
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/adechiaro/.ssh/id_rsa):
   Enter passphrase (empty for no passphrase):
   Enter same passphrase again:
   Your identification has been saved in id_rsa.
   Your public key has been saved in id_rsa.pub.
   The key fingerprint is:
   5d:34:f1:de:ad:be:ef:65:34:36:a4:0d:75:d6:3c:47 adechiaro@desktop
   The key's randomart image is:
   +--[ RSA 2048]----+
   ...
   +-----------------+
   (adechiaro@desktop:pts/6)-(4/0 @ 17k)-(9:22 PM)
   ~$ cat .ssh/id_rsa.pub
   ssh-rsa <key> adechiaro@desktop

2. Any machine you want to be able to connect to with this key, login and copy the contents of your public key (id_rsa.pub) to the authorized_keys file.  These are all in your $HOME/.ssh/ directory.  There are various ways to do this: you could copy the file over with scp and cat/append it, you could remote in to the host and cut & paste the data, if you had a large infrastructure you could use ssh-copy-id or similar custom script.  It’s up to you, something like what is below should work in the general case.  Also the <key> is your public key in base64 encoded format.

   desktop:~$ scp .ssh/id_rsa.pub adechiaro@server:~
   id_service.pub                           100%  399     0.4KB/s   00:00
   desktop:~$ ssh adechiaro@server
   server:~$ cat id_rsa.pub >> ~/.ssh/authorized_keys

3. Now for an example of making the key more secure, you can add additional options to the authorized_keys file.  These come before the “ssh-rsa <key>” part of the entry (prefix the line with them): Read the rest of this entry »

Just thought it might be useful posting the contents of my .screenrc file. I’ll go over it in some basic detail.

escape ^||
msgwait 2

altscreen on
autodetach on
defscrollback 4000
startup_message off

# Message/bell/activity info
vbell off
vbell_msg " *beep* "
sorendition "+b kG"
bell "%c:bell -> %n%f %t^G"
activity "%c activity -> %n%f %t"

# Don't block screen session waiting for unresponsive window
nonblock on

hardstatus alwayslastline '%{= gk}%-Lw%{= rW}%50> %n%f* %t %{-}%+Lw%< %= %{= kG} %H %{= Bw} %l %{= kG} %Y-%m-%d %c %{g}' 

# Window number starts at 1, not 0
bind 'q' quit
bind c screen 1
bind 0 select 10

# Tell screen that you term can scroll and bind Shift+PgUp/PgDn
termcapinfo xterm ti@:te@
bindkey -m "^[[5;2~" stuff ^b
bindkey -m "^[[6;2~" stuff ^f

# Setup our default apps
screen -t htop 1 htop
screen -t iptraf 2 nice sudo iptraf
screen -t messages -M 3 tail -n 2000 -F /var/log/messages
screen 4 bash
select 1

OK, so firstly I like using Ctrl-Pipe for some reason – ‘^|’  I set some initial startup parameters, autodetach useful in case you need to kill X for some reason.  Then is the bell & activity customizations, nothing special to look at here.  My hardstatus is something I threw together based upon several I’ve seen online and the custom one I wrote for work.  I configured it to start numbering with ‘1′ instead of ‘0′ for window numbers and for xterm to recognize scrolling with my mousewheel.  Final section is just setting up some default windows.  I found the ‘-F’ flag to be vital versus the typical lowercase ‘-f’, as it implies ‘–retry’.  This is needed due to log rotation, it will force tail to retry opening file upon failure.  Otherwise once the file gets rotated, output from tail will just stop.

Screenshots:

This question stumped me for awhile several years back, at the time I was working from home and my ISP would block SMTP relaying to any host other then their own (as most typically do).  I already had a free Dynamic DNS account with DynDNS  and decided to add their inexpensive Mailhop Outbound service which worked great and solved all my problems for a whole $15/year*.  Thinking about it more I figured it would be make sense to utilize it further and have my Linux server relay any emails to my cellphone/Blackberry, especially any mdadm monitoring alerts for my RAID-5 array.  However I was having difficulting locating quality documentation configuring relayhost SMTP authentication in sendmail.  It’s a simple setup now with Postfix (and probably sendmail too), but at the time I was running Fedora Core and just switched to the brand-new Warty release (I wasn’t kidding when I said some time ago).

Here are some basic configuration steps.  We will be using mail.myrelayhost.com and testing delivery to bob@aol.com as an example.  I have not tested this with DynDNS relaying in some time now but I see no reason that would prevent it from working if you use their service.  Note: These commands must be run with root permissions via sudo.

Read the rest of this entry »

When I originally build my NFS/NAS server I decided to go for a software-based RAID 1 array for the OS install itself.  The storage drive was RAID-5 based but that is a different story.  I can’t recall the specifics exactly but I do remember a few times having difficulty booting off of the second drive and during the rebuild thought a slightly different design would be worth a try.  Since I had two identical drives from the array I figured the mirroring was a good idea, it was just the mdadm driver that I wanted to avoid.  Instead a manual mirror with rsync was in order.

I did base my design upon some very good backup reference information.  Plus with this way it would be easy to boot from if ever needed and take tar archives of the data without dealing with file consistency or modification issues.

Here’s how it works:

Read the rest of this entry »