Anthony’s Blog

Overall I am fairly impressed with Intrepid.  They seemed to have fixed a number of bugs I encountered (especially that annoying menu pulldown delay with dual screen configurations), and I’ve actually switched back to full compiz mode.  Most things work fairly smoothly and performance seems a bit better as well.  There is one strange bug I’ve found where there is a vertical black bar on the right-hand side of my 2nd monitor, which cuts off the gnome-panel at the top of my screen.  What doesn’t make much sense to me is that maximized windows do not extend over into the dark limbo, unlike the panel:

Odd desktop/resolution bug

Strange.  Maybe it didn’t auto-detect my old 2005FPW right, pretty sure I had a manually configured xorg.conf just prior to the upgrade  I’ll have to look into it further.

Regardless, now it comes time to design/implement more items that have been held back too long.  Firstly will entail setting up a basic SSH key repository for my various machines and accounts.  There are a number of good articles about this topic.  What I find particularly useful are the ‘from’ and ‘command’ options within OpenSSH (referenced in the last two URL’s).  I have been wondering about a solution to this for some time; I wanted to create passphrase-less SSH keys used for automation & scripting but did not want full access to anyone who may have compromised the keys.  This solution seems like it will fit perfectly!  Already have a few sample ideas to use this with.

After setting up SSH keys, next will come full NIS implementation.  I already have it installed on my server, but not configured or running at the current time.  NFS-mounted home directories always seem like a good option, and I would normally agree but I am not sure if I want the usability of my desktop system tied dependent upon the server being up and running.  Right now if the server goes down, I can still function fine on my desktop – just without access to my RAID storage array.  Also due to come will be a post on performance logging & monitoring, I love my GkrellM but something with metrics storage would be nice.

Just thought it might be useful posting the contents of my .screenrc file. I’ll go over it in some basic detail.

escape ^||
msgwait 2

altscreen on
autodetach on
defscrollback 4000
startup_message off

# Message/bell/activity info
vbell off
vbell_msg " *beep* "
sorendition "+b kG"
bell "%c:bell -> %n%f %t^G"
activity "%c activity -> %n%f %t"

# Don't block screen session waiting for unresponsive window
nonblock on

hardstatus alwayslastline '%{= gk}%-Lw%{= rW}%50> %n%f* %t %{-}%+Lw%< %= %{= kG} %H %{= Bw} %l %{= kG} %Y-%m-%d %c %{g}' 

# Window number starts at 1, not 0
bind 'q' quit
bind c screen 1
bind 0 select 10

# Tell screen that you term can scroll and bind Shift+PgUp/PgDn
termcapinfo xterm ti@:te@
bindkey -m "^[[5;2~" stuff ^b
bindkey -m "^[[6;2~" stuff ^f

# Setup our default apps
screen -t htop 1 htop
screen -t iptraf 2 nice sudo iptraf
screen -t messages -M 3 tail -n 2000 -F /var/log/messages
screen 4 bash
select 1

OK, so firstly I like using Ctrl-Pipe for some reason – ‘^|’  I set some initial startup parameters, autodetach useful in case you need to kill X for some reason.  Then is the bell & activity customizations, nothing special to look at here.  My hardstatus is something I threw together based upon several I’ve seen online and the custom one I wrote for work.  I configured it to start numbering with ‘1′ instead of ‘0′ for window numbers and for xterm to recognize scrolling with my mousewheel.  Final section is just setting up some default windows.  I found the ‘-F’ flag to be vital versus the typical lowercase ‘-f’, as it implies ‘–retry’.  This is needed due to log rotation, it will force tail to retry opening file upon failure.  Otherwise once the file gets rotated, output from tail will just stop.

Screenshots:

Well, the upgrade to Intrepid went smoothly during the install process itself.  However after a reboot, the system hung partially through boot and dropped to a initramfs shell claiming “cannot find root device /dev/disk/by-uuid/50128bb8…” and “Gave up waiting for root device.”  Wonderful.  Tinkered around a bit, tried mounting drives manually only they were not listed in /dev.  Attempted booting old 2.6.24-18 kernel which worked fine.  Aha, so it’s something related to new kernel.  Did a quick search which revealed the following bug:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/290153

Apparently on certain hardware the kernel has a bug which causes a long timeout for the SCSI/SATA bus.  It took a good 2-3 minutes on my system but when I left it idle while I was reading the bug report on my desktop system, a bunch more lines flew by from the initramfs prompt about ata bus reset and detecting new drives.  After that a simple ‘exit’ from prompt continued a normal boot.

It’s a fairly important bug but at least a workaround exists.  I’ve tinkered with adding the ‘rootdelay’ option to my menu.lst but have not found the best match yet.  Maybe I’ll just leave it as is, my server almost never gets rebooted.  You’re instilling me with a lot of confidence doc, I mean Intrepid.  Definitely going to make a full backup of my desktop machine before attempting upgrade on that one.

So I am going through the process of upgrading my server to 8.10.  A quite useful HOWTO on howtoforge.com can be found guiding through the process (they also document upgrading from Desktop version as well).

I was not sure which exact command to run given that my headless server obviously doesn’t have update-manager running.  The HOWTO covers usage of the ‘do-release-upgrade’ command.  Only thing I ran beforehand was my rootfs rsync script to make a backup copy of my OS drive incase the worst happen.

If this runs smoothly I will make a backup copy of my desktop rootfs drive and do a similar upgrade to Intrepid.  I am already aware of one or two things I’m not keen on with Intrepid, notably that btnx is not compatible!  For those not aware, btnx was the premier application for configuring and making use of every single one of those buttons on the higher-end mice.  I have a Logitech MX Laser something and have it set up perfectly, tilt wheel left/right for forward/back in Firefox, extra buttons for minimize or close windows (Ctrl-W), etc.  I spend weeks trying to get it working the way I wanted with xmodmap and that ended in nothing but frustration.  I’m sure there will be some other things that don’t work quite the way I would like so a mirrored backup drive pre-upgrade is nice to have.

Finally!   After awaiting a solution for sometime to be released, Ibex will  have kernel 2.6.26 which  supports read-only bind mounts.  I discovered this as a rather serious security breach to my proposed system design  some time ago – I was trying to implement remote access for myself and friends to my data storage.  I figured some form of SSH-based access would be a good start, but I didn’t want to have any accounts directly open on my server or desktop machine.  Since building separate physical hardware just for this would be a waste of resources, I thought the best solution would be a virtual machine.  Configuring NFS on it could potentially be another security hole (not to mention more overhead then needed), I knew a bind mount would be perfect – a read-only one of course.  However as I was testing it I quickly realized that read-only bind mounts weren’t actually read-only.  Thus, the problem.  I suppose since I keep most of my multimedia files marked immutable it wouldn’t be a real problem unless someone got root.  Still rather be safe then sorry.  More to follow about this later.

I read over a few postings to the kernel mailing list which addressed this last year, but this was just in the development phase then.  The solution the kernel architects created involved updating the VFS code, since all bind mounts are implemented in the virtual layer.  You can read more of the technical aspects over at lwn.net.

This question stumped me for awhile several years back, at the time I was working from home and my ISP would block SMTP relaying to any host other then their own (as most typically do).  I already had a free Dynamic DNS account with DynDNS  and decided to add their inexpensive Mailhop Outbound service which worked great and solved all my problems for a whole $15/year*.  Thinking about it more I figured it would be make sense to utilize it further and have my Linux server relay any emails to my cellphone/Blackberry, especially any mdadm monitoring alerts for my RAID-5 array.  However I was having difficulting locating quality documentation configuring relayhost SMTP authentication in sendmail.  It’s a simple setup now with Postfix (and probably sendmail too), but at the time I was running Fedora Core and just switched to the brand-new Warty release (I wasn’t kidding when I said some time ago).

Here are some basic configuration steps.  We will be using mail.myrelayhost.com and testing delivery to bob@aol.com as an example.  I have not tested this with DynDNS relaying in some time now but I see no reason that would prevent it from working if you use their service.  Note: These commands must be run with root permissions via sudo.

Read the rest of this entry »

I was taking a mid-afternoon nap (yes at 3 am, I work nights) and I came back to my PC to see CPU usage on my server hovering around 15% – not at idle like usual.  Doing a quick check revealed md0_raid5 and md0_resync running which is normally not a good sign.

mdadm –detail /dev/md0 showed the following:

    Update Time : Sun Oct  5 03:22:19 2008
          State : clean, recovering
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
         Layout : left-symmetric
     Chunk Size : 64K
 Rebuild Status : 85% complete

Uh oh.  Why was the array rebuilding itself?  All drives were listed as active and working …  but did we experience a drive momentarily dropping from the array or a SATA device reset?  Was this a sign of impending hardware failure?  Tailing /var/log/messages displayed this useful piece of information:

Oct  5 01:06:01 rigel md: data-check of RAID array md0

Ok, so “data-check” doesn’t sound so worrysome.  A quick Google search revealed this nice gem:

root@rigel:~# tail /etc/cron.d/mdadm
# By default, run at 01:06 on every Sunday, but do nothing unless the day of
# the month is less than or equal to 7. Thus, only run on the first Sunday of
# each month. crontab(5) sucks, unfortunately, in this regard; therefore this
# hack (see #380425).
6 1 * * 0 root [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ] && /usr/share/mdadm/checkarray --cron --all --quiet

Ah, so this is the first Sunday of the month and the check kicked off at 1:06 AM.  You trixies Ubuntu.  Apparently a bug has been filed causing performance issues on some boxes.  Good idea to verify data integrity, although slightly more obvious notice would be nice.

When I originally build my NFS/NAS server I decided to go for a software-based RAID 1 array for the OS install itself.  The storage drive was RAID-5 based but that is a different story.  I can’t recall the specifics exactly but I do remember a few times having difficulty booting off of the second drive and during the rebuild thought a slightly different design would be worth a try.  Since I had two identical drives from the array I figured the mirroring was a good idea, it was just the mdadm driver that I wanted to avoid.  Instead a manual mirror with rsync was in order.

I did base my design upon some very good backup reference information.  Plus with this way it would be easy to boot from if ever needed and take tar archives of the data without dealing with file consistency or modification issues.

Here’s how it works:

Read the rest of this entry »