Since then I’ve been patiently waiting for the public launch, and last night I spent watching the live webcast and finally playing around with the engine.  It is still very much in a beta stage as it can only understand certain branches of knowledge.  But for what it can do, wow.  I’m extremely interested to learn how Wolfram exactly accomlished all this, I understand it uses Mathematica as a backend but just the idea of expressing that depth and breath of data in a computing language is fascinating.  There were a few clips of the engineers talking about the infrastructure.  Hardware geeks would get their fill at the supercomputer they build to run this thing (44th largest @ 10,000 cores using Dell quad Xeon’s and nearly an exabyte of storage).  Oh, and they’ve opened up an API for developers!

There are lots of example queries to browse through, but here are some simple ones:

Weather on a particular date – http://www.wolframalpha.com/input/?i=weather+november+5+1955

…no thunderstorms predicted that night at 45 F and overcast (uses your current location).

The natural-language parser is fairly flexible.  You can enter queries like “weather day obama was born”.  It’s far from perfect and chokes on more complicated strings but useful nonetheless.

Playing around with the knowledge engine for a few minutes, I’ve learned that:

• A 5 earth mass body orbiting a 10 solar mass star with a semi-major axis of 2.5 AU has a period of 1.251 years.
• A 50 megaton explosion (TNT) is:
  • 1.2 times the total energy that hits earth every second from the sun
  • 1.0 times the energy released the Krakatoa eruption and the amount of energy
  • Has the same energy as a relativistic mass of 2.3 kg
• Hurricane Andrew lasted 4 days longer then Katrina, but had the same maximum wind speed of 150 mph (on dates 5 calendar days apart)

You can also get nice visual representations of chords.  Or checkout the blackbody spectrum at the temperature of the surface of the sun.

Go give it a shot!  You can also download W|A toolbars, firefox search engine add-ons, gadgets, and more.

Now came the process of migrating the data over from my file server to the new array.  I didn’t want to fiddle with NFS or anything of that nature since I was booting off a Jaunty live CD.  SSH was certainly possible but then there was the overhead.  Rsync was another solution.  But then I recalled a extremely fast way with tar & netcat.  Combine that with pipeviewer and you’ll get a fast transfer complete with verbose information.

It’s very simple.  With two computers, source and dest, you simply run the following:

On source (with IP 192.168.0.10):

tar -cf - /path | pv | nc -l -p 8888 -q 5

This will tar up /path and pipe it to netcat, which is listening on port 8888 for an incoming connection.

On dest:

nc 192.168.0.10 | pv | tar -xf -

Connect to source and pipe the output to tar for extraction.

Credit to Peteris’ wonderful blog entry on pv.

If you’re unfamiliar with the term “autodidact”, you’re not alone. An autodidact is an individual who takes the initiative to teach themselves, rather than go through the formal process of education and studying under a professor. Autodidactism, as it is known, has a long history and includes such luminaries as Socrates, Benjamin Franklin, and Thomas Edison. Even Samuel Clemens once famously wrote as Mark Twain, “I have never let my schooling interfere with my education”. Indeed.

I don’t believe autodidact sounds familiar, but definitely something I’m familiar with.  The way I see it, it’s the only way to truly know a subject.  Maybe you will catch some inspiraction reading the rest of Matt’s posting.

On a related note, I’ve considered tinkering with creating my own local repository mirror.  Not that I have nearly enough machines to make it necessary, but it would be an entertaining exercise.  Even found a basic HOW-TO or two.  However I have heard of potential issues: it can take weeks to fully mirror several distributitions (several GB each) and with an incomplete repository it would be somewhat pointless to use.  Luckily there seems to be an easy solution with mod_rewrite.

Firstly, it will likely come pre-installed with Ubuntu Jaunty Beta, although possibly Gentoo.  Both can run on the plug in addition to FreeBSD.  Wiki explains more.  It also has a good entry on configuring the plug to boot from an SD card rather then the onboard NAND flash.

The forums are a very useful resource as well:

Configuring the USB serial console (Linux & Windows).  My Ubuntu desktop failed to create a /dev/ttyUSB1 on it’s own when plugged in, a simple ‘modprobe ftdi_sio product=0×9e8f vendor=0×9e88′ resolved that.

Give care when flashing partitions or images and make sure you don’t overwrite /dev/mtd0 or /dev/mtdblock0 (1st MB of NAND), it holds the uBoot boot image, the plug cannot boot without it.  Next partition is the Linux kernel image (~1-4mb), and finally a root filesystem image (remainder of the 512MB).

I somehow managed to brick mine at one point, after a fsck and reboot I got a “Verifying Checksum … Bad Data CRC” error and it would not boot.  Playing with it a bit, I managed to get it to boot by tftp’ing a kernel image and mount a root filesystem via NFS.  GST’s documentation and the wiki provide plenty of guidance.  I also had issues getting it to re-flash the kernel image to /dev/mtd1 because the kernel image provided by GST was slightly too big for the default size of mtd1.  Since then I’ve read they have released a new kernel image that fits, but at the time I simply increased the size of mtd1 (to 4mb rather then the default 1mb) by editing the mtdparts setting in uBoot.

It also occured to me that coming up with a makeshift bootmenu might be useful for debugging:

set console 'console=ttyS0,115200 mtdparts=nand_mtd:0x00100000@0x00000000(uBoot)ro,0x00400000@0x00100000(uImage),0x1fb00000@0x00500000(rootfs)'

set boottftp 'tftpboot 0x2000000 $(image_name)'
set bootargs_nfs 'root=/dev/nfs rw'
set boot_nfs 'run boottftp;setenv bootargs $(console) $(bootargs_nfs) nfsroot=$(serverip):$(rootpath) ip=$(ipaddr):$(serverip)$(bootargs_end);bootm 0x2000000'

set bootnand 'nand read.e 0x2000000 0x00100000 0x00500000'
set bootargs_nand 'root=/dev/mtdblock2 rw'
set boot_nand 'run bootnand;setenv bootargs $(console) $(bootargs_nand) ip=$(ipaddr):$(serverip)$(bootargs_end);bootm 0x2000000'

set bootargs_sd 'root=/dev/mmcblk0p1 rw'
set boot_sd 'run bootnand;setenv bootargs $(console) $(bootargs_sd) ip=$(ipaddr):$(serverip)$(bootargs_end);bootm 0x2000000'

set bootcmd 'run boot_sd;'

As I mentioned, the mtdparts I provided are configured for a 4mb mtd1 partition, you may need to modify this.  This thread explains mtdparts better.  All of the other variables such as IP, SERVERIP, etc need to be configured in addition to these listed here.  You will also see it’s set to boot from SD card by default, it can be changed to ‘run boot_nand’ or ‘run boot_nfs’ if you like.

My Linux-based SheevaPlug (embedded Linux in a wall-wart) has been occupying some time as well, tinkering with the Jaunty install and NFS and SSH.  I’m pretty impressed with the capabilities of this so far, despite the fact I managed to partially brick it for awhile. I eventually want to build a Tweet-A-Watt and use my plug for interfacing.  Wiring my 1-wire weather station to the plug makes sense as well.

Apticron is very easy to set up, it’s in the repositories and requires basically no configuration.  It will drop a script in /etc/cron.daily and that is about it, emailing any reports to root.  Of course this can be modified through a .forward or an entry in /etc/aliases.

Logcheck is fairly simple to set up as well – it is also in the repositories.  Once installed, edit the /etc/logcheck/logcheck.conf file to configure.  The first thing you will want to set is the REPORTLEVEL setting, options are “workstation”, “server” (default value), or “paranoid”.  I use server on mine, which gives a good amount of detail. I would advise against using paranoid unless the server is extremely locked down and users do not typically login.  Workstation is good for a desktop environment.  The only other variable I edited was SENDMAILTO.  Logcheck works by basically comparing each  logentry against a set of regular expressions and generate a report if it does not match.  I had to modify one or two regex’s slightly to fix false positives, if you want my changes just ask and I’ll send them over.

One other small gem I want to mention : gkrellm.  I use this on both my desktop and server, it is invaluable for providing real-time system performance metrics.  Sure, it does not have any logging capabilities and thus unsuitable in a large-scale environment but for keeping an eye on one or two boxes it fits the bill quite nicely.

So it’s been some time since I posted, unfortunately life has kept me on the go these past several weeks and I’ve had limited time at my desk.  However, thankfully I was able to work on Ktux from my netbook and now I successfully have multitasking working!  I attempted several previous implementations of a scheduler without success, this time I re-wrote mostly everything from scratch, minus a few lines of the timer ISR assembly code.  Currently, the scheduler supports thread creation/destruction and basic priority adjustment with nice() (just adjusts the process quantum timeslice).  Oh, and a simple yield() was implemented as well.  As of the moment, I have a pointer array to keep track of the threads, but in the process of replacing with linked lists (same implementation Linux kernel uses).

The screenshot below demonstrates 4 threads of execution, 3 of them increment counters (with various priorities) and the final one just displays the values on the screen, then yields():

Now, of course with with multitasking support comes a whole new array of synchronization issues.  I have intentionally followed the KISS principle thus far when developing the kernel framework, knowing that any substantial code would most likely need to be re-written with critical sections, locks, etc.  Due to the complexities and time involved, I am fairly certain I will not aim for full SMP support in Ktux.  A few basic atomic operations have been written, but for the moment I think I will just use something similar to the big kernel lock in Linux; performance is not my main concern here, getting something out the door that actually runs is.

Not sure where I will go from here, I know the virtual memory needs a *LOT* of work, it’s pretty ugly right now.  From there, I think user-level thread support will be a logical next step.