Anthony’s Blog

So after installing Intrepid on my new new Acer Aspire One netbook, I was working with SMART to resolving the Load_Cycle_Count issue (hard drive killer bug).  Once that was done I figured it would make sense to check the SMART health status of the rest of the drives in my desktops and server.  Good thing I did too…  results varied to a few counts of UDMA CRC Error Count to having hundreds of thousands of Hardware ECC Recovered, Raw Read Error Rate, and Seek Error Rates.  The RAID drives in my server had a few million counts of Seek Error Rate as well.  However I did run self-tests and all came back passed so it is something I am going to keep a close eye on (GkrellM plugin perhaps?)

According to a few different sources there are a few of the attributes which are important to watch, although opinions can vary slightly.  You can also get descriptions of each attribute here.  Also keep in mind I’m taking RAW_VALUE here, this Linux Journal article helps explain the difference between VALUE/WORST/THRESH/RAW values, however the data can be a little difficult to interpret (I usually stick with RAW).  Basically the best idea here is to keep current backups (a whole book can be written about those alone) and to schedule regular SMART self-tests for all your drives.  I used the following configuration in /etc/smartd.conf:

/dev/sda -a -o on -S on -s (S/../.././12|L/../../3/1) -m root
/dev/sdb -a -o on -S on -s (S/../.././12|L/../../3/3) -m root

This will configure regular ’short’ scans at noon daily, and ‘long’ scans at 1 and 3 pm on Wednesdays.  Other parameters provide more verbose and frequent information (-a) and enabling autosave (-S) and automatic offline (-o).  -m simply tells to mail any warnings (if found) to root.  -M will do the same but send all reports instead of just warnings.

Update (Dec 05): Note – this is important.  Depending upon your distribution, you may need to modify your /etc/default/smartmontools file.  On my Intrepid boxes I needed to uncomment the line “start_smartd=yes”.  Without it, smartd would not start up automatically and furthermore any attempts to start it would silently fall back to the prompt without any error message.  I was wondering why my automated selftests were not being run…

So this is certainly nothing new, but I’ve been unimpressed with the changes del.icio.us made to their interface back in this summer.  They used to have the original bookmark post date stored in a hidden FORM variable when editing in “full-screen” mode.  It was handy to edit so I could sort the bookmarks in my live bookmark RSS feeds.  Now it’s impossible to do so, not fun.  A Google search revealed next to nothing, asides from this one blog entry.  Suggestions would be appreciated.

Overall I am fairly impressed with Intrepid.  They seemed to have fixed a number of bugs I encountered (especially that annoying menu pulldown delay with dual screen configurations), and I’ve actually switched back to full compiz mode.  Most things work fairly smoothly and performance seems a bit better as well.  There is one strange bug I’ve found where there is a vertical black bar on the right-hand side of my 2nd monitor, which cuts off the gnome-panel at the top of my screen.  What doesn’t make much sense to me is that maximized windows do not extend over into the dark limbo, unlike the panel:

Odd desktop/resolution bug

Strange.  Maybe it didn’t auto-detect my old 2005FPW right, pretty sure I had a manually configured xorg.conf just prior to the upgrade  I’ll have to look into it further.

Regardless, now it comes time to design/implement more items that have been held back too long.  Firstly will entail setting up a basic SSH key repository for my various machines and accounts.  There are a number of good articles about this topic.  What I find particularly useful are the ‘from’ and ‘command’ options within OpenSSH (referenced in the last two URL’s).  I have been wondering about a solution to this for some time; I wanted to create passphrase-less SSH keys used for automation & scripting but did not want full access to anyone who may have compromised the keys.  This solution seems like it will fit perfectly!  Already have a few sample ideas to use this with.

After setting up SSH keys, next will come full NIS implementation.  I already have it installed on my server, but not configured or running at the current time.  NFS-mounted home directories always seem like a good option, and I would normally agree but I am not sure if I want the usability of my desktop system tied dependent upon the server being up and running.  Right now if the server goes down, I can still function fine on my desktop – just without access to my RAID storage array.  Also due to come will be a post on performance logging & monitoring, I love my GkrellM but something with metrics storage would be nice.

Just thought it might be useful posting the contents of my .screenrc file. I’ll go over it in some basic detail.

escape ^||
msgwait 2

altscreen on
autodetach on
defscrollback 4000
startup_message off

# Message/bell/activity info
vbell off
vbell_msg " *beep* "
sorendition "+b kG"
bell "%c:bell -> %n%f %t^G"
activity "%c activity -> %n%f %t"

# Don't block screen session waiting for unresponsive window
nonblock on

hardstatus alwayslastline '%{= gk}%-Lw%{= rW}%50> %n%f* %t %{-}%+Lw%< %= %{= kG} %H %{= Bw} %l %{= kG} %Y-%m-%d %c %{g}' 

# Window number starts at 1, not 0
bind 'q' quit
bind c screen 1
bind 0 select 10

# Tell screen that you term can scroll and bind Shift+PgUp/PgDn
termcapinfo xterm ti@:te@
bindkey -m "^[[5;2~" stuff ^b
bindkey -m "^[[6;2~" stuff ^f

# Setup our default apps
screen -t htop 1 htop
screen -t iptraf 2 nice sudo iptraf
screen -t messages -M 3 tail -n 2000 -F /var/log/messages
screen 4 bash
select 1

OK, so firstly I like using Ctrl-Pipe for some reason – ‘^|’  I set some initial startup parameters, autodetach useful in case you need to kill X for some reason.  Then is the bell & activity customizations, nothing special to look at here.  My hardstatus is something I threw together based upon several I’ve seen online and the custom one I wrote for work.  I configured it to start numbering with ‘1′ instead of ‘0′ for window numbers and for xterm to recognize scrolling with my mousewheel.  Final section is just setting up some default windows.  I found the ‘-F’ flag to be vital versus the typical lowercase ‘-f’, as it implies ‘–retry’.  This is needed due to log rotation, it will force tail to retry opening file upon failure.  Otherwise once the file gets rotated, output from tail will just stop.

Screenshots:

Well, the upgrade to Intrepid went smoothly during the install process itself.  However after a reboot, the system hung partially through boot and dropped to a initramfs shell claiming “cannot find root device /dev/disk/by-uuid/50128bb8…” and “Gave up waiting for root device.”  Wonderful.  Tinkered around a bit, tried mounting drives manually only they were not listed in /dev.  Attempted booting old 2.6.24-18 kernel which worked fine.  Aha, so it’s something related to new kernel.  Did a quick search which revealed the following bug:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/290153

Apparently on certain hardware the kernel has a bug which causes a long timeout for the SCSI/SATA bus.  It took a good 2-3 minutes on my system but when I left it idle while I was reading the bug report on my desktop system, a bunch more lines flew by from the initramfs prompt about ata bus reset and detecting new drives.  After that a simple ‘exit’ from prompt continued a normal boot.

It’s a fairly important bug but at least a workaround exists.  I’ve tinkered with adding the ‘rootdelay’ option to my menu.lst but have not found the best match yet.  Maybe I’ll just leave it as is, my server almost never gets rebooted.  You’re instilling me with a lot of confidence doc, I mean Intrepid.  Definitely going to make a full backup of my desktop machine before attempting upgrade on that one.

So I am going through the process of upgrading my server to 8.10.  A quite useful HOWTO on howtoforge.com can be found guiding through the process (they also document upgrading from Desktop version as well).

I was not sure which exact command to run given that my headless server obviously doesn’t have update-manager running.  The HOWTO covers usage of the ‘do-release-upgrade’ command.  Only thing I ran beforehand was my rootfs rsync script to make a backup copy of my OS drive incase the worst happen.

If this runs smoothly I will make a backup copy of my desktop rootfs drive and do a similar upgrade to Intrepid.  I am already aware of one or two things I’m not keen on with Intrepid, notably that btnx is not compatible!  For those not aware, btnx was the premier application for configuring and making use of every single one of those buttons on the higher-end mice.  I have a Logitech MX Laser something and have it set up perfectly, tilt wheel left/right for forward/back in Firefox, extra buttons for minimize or close windows (Ctrl-W), etc.  I spend weeks trying to get it working the way I wanted with xmodmap and that ended in nothing but frustration.  I’m sure there will be some other things that don’t work quite the way I would like so a mirrored backup drive pre-upgrade is nice to have.

Finally!   After awaiting a solution for sometime to be released, Ibex will  have kernel 2.6.26 which  supports read-only bind mounts.  I discovered this as a rather serious security breach to my proposed system design  some time ago – I was trying to implement remote access for myself and friends to my data storage.  I figured some form of SSH-based access would be a good start, but I didn’t want to have any accounts directly open on my server or desktop machine.  Since building separate physical hardware just for this would be a waste of resources, I thought the best solution would be a virtual machine.  Configuring NFS on it could potentially be another security hole (not to mention more overhead then needed), I knew a bind mount would be perfect – a read-only one of course.  However as I was testing it I quickly realized that read-only bind mounts weren’t actually read-only.  Thus, the problem.  I suppose since I keep most of my multimedia files marked immutable it wouldn’t be a real problem unless someone got root.  Still rather be safe then sorry.  More to follow about this later.

I read over a few postings to the kernel mailing list which addressed this last year, but this was just in the development phase then.  The solution the kernel architects created involved updating the VFS code, since all bind mounts are implemented in the virtual layer.  You can read more of the technical aspects over at lwn.net.