ZeroMQ libraries & stability / radarwatch update

I’ve been hacking with ZeroMQ for some time now…  reading through the excellent guide and working on the preliminary steps to incorporate it into the new version (migrated to GitHub from Google Code) of my Doppler radar analysis application.  Multi-threading support was on the list of original criterion and considering many of the other analysis features I’d like to eventually add, full distributed processing would be the best design.  I’m not an expert hand at SMP application design and although the implementation needs here would be fairly trivial, choosing a good library to handle the nasty bits would make things vastly easier.

So I tinkered with several of their samples designs, implemented a few of the patterns in my code and tested things out.  Initial results with just a simple multithreaded service seemed work OK at first glance but eventually dropped data caused issues so I decided to try something with reliability such as the paranoid pirate pattern (robust reliable queuing).  Thinking back, I should have realised something was up when it exhibited that issue on both inproc IPC and tcp/localhost.  No matter what I was trying to do porting examples into my code nothing remained stable; either it crashed immediately or processed a few messages then died.  C and C++ code showed similar results.  Running the examples on their own worked fine.  The front-end/client-side Python application worked fine.  I gave it a fresh shot with another example – the Majordomo pattern (service-oriented reliable queuing) again showed similar symptoms of instability.  Commenting out huge sections of my code thinking it may of had something to do with a broken pointer or some other oversight on my part had little to no results.  Eventually I started comparing the GCC invocations and noticed there were some minor differences.  After 5 minutes of testing the weeks of tinkering finally yielded some fruit… Google confirmed my suspicions: linking ZeroMQ with -pg for gprof support crashes the library.

It’s amazing how quick it can be to find what you’re looking for when you know exactly what you want.  At least now I can get back to implementing 0MQ in my code.

More sensor interfacing with the Bus Pirate

After interfacing the TMP102 temperature sensor recently with the Bus Pirate, the next step on my agenda was to integrate this with the Web Platform along with a HH10D humidity sensor.  Unfortunately I realised a bit late (entirely my fault rushing an order) that although the HH10D was an I2C sensor, the humidity was actually read through a frequency output pin; the I2C EEPROM was only for calibration data.  Bus Pirate to the rescue once again!

This, like the TMP102 is an 3.3V sensor and needs pullup resistors on the I2C bus to read the EEPROM.  I wasted spent a number of hours with this getting nothing but garbage data with the first sensor I tried (the calibration values were completely wrong), on the second try with a backup sensor I had much better luck, getting good results almost immediately.

We’ll be doing a similar setup to the TMP102: using I2C and turning on the power supplies but in this case the AUX pin should be set to HIGH-Z for input/reading.

HiZ> m
1. HiZ
2. 1-WIRE
4. I2C
5. SPI
6. 2WIRE
7. 3WIRE
9. LCD
x. exit(without change)

(1)> 4
Set speed:
 1. ~5KHz
 2. ~50KHz
 3. ~100KHz
 4. ~400KHz

(1)> 3
I2C> c
a/A/@ controls AUX pin
I2C> @
I2C> W
Power supplies ON
I2C> (1)
Searching I2C address space. Found devices at:
0xA2(0x51 W) 0xA3(0x51 R)

Good.  Now if you take a look at the datasheet for the M24C02 EEPROM to do a random address read you’ll need to send a start bit, device select (read only; R/W set to 0), address, then another start bit, device select again (but with R/W set to 1), then read commands.  The HH10D datasheet mentions there are 2 calibration factors we need to read: sensitivity and offset and each are 2 byte integers, making a total of 4 bytes.  These start at address EEPROM address 10.  Using the BP, we do the following:

I2C> [0xa2 0x0a [0xa3 r:4
READ: 0x01  ACK 0x8B  ACK 0x1E  ACK 0x32

This is in big-endian format so the sensitivity value (at address 0x10) reads 0x018B and the offset is 0x1E32.  We’ll also do a frequency read, the first as baseline with my AC running nearby and the second breathing on the sensor:

I2C> f
AUX Frequency:  autorange 7,169 Hz
I2C> f
AUX Frequency:  autorange 6,790 Hz

The formula in the datasheet states the RH = (offset-freq) * sensitivity/4096.  Plugging these in we get 54% and 90.6% RH which seems to match the other humidity sensor in my room.  Success!

Next step was getting this working on my Web Platform.  I have some very bare-bones code working with the TMP102 already, just need to work on the HH10D a bit more and I’ll post my example up here.

Hard drive rotary input device part 2

Been working on a few projects lately and have a number of things I’m hoping to post soon, one of them I got around to briefly documenting is the re-implementation of a hard drive rotary input device – I’m working on hooking up a laptop hard drive to an Arduino/AVR to be used as an input device similar to a rotary encoder only super-sized.  I originally put together a circuit last year which never really worked but now I’ve gotten around to doing it right.  The circuit I implemented was similar to the original Instructable with a few modifications, I tried using the LM324 op-amp called for but didn’t quite work the way I wanted (running from +5V Vcc gave an output ~3.5V instead of being saturated to the full 5V) and tried using a LM339 comparator instead with better success.  The signals can be seen for comparison on my scope in the video.  Right now this just proof-of-concept generating the correct inputs signals to the microcontroller.  Implementation of firmware, a simple circuit to the AVR and brief demo of a more sophisticated output display will follow shortly. Details of the circuit are on the YouTube page and listed below.

The schematic I followed was similar to the one in the Instructable however I removed the 1k series resistors to the non-inverting inputs of the ‘324 and replaced the 10k feedback with 1M’s. The outputs of the LM324 were fed to the LEDs with a series current-limiting resistor. The 339 was similar except it had a 10k pull-up resistor on the output. Both were running off of a single-sided +5V supply and the inverting inputs were tied to ground. The hard drive I used had 4 pins as shown, 3 pins each to separate coils and one common pin which was also tied to ground.

Barometer OneWire sensor added

I finally got around to ordering and building a 1-wire barometer kit from Hobby Boards…  it was a fairly straightforward assembly process when you actually follow the construction tips and don’t accidentally mix up the location of the two SMT IC chips…  Oops!  Thankfully they were only 8-pin SOIC’s so it wasn’t too terribly difficult to fix.  Next time I suppose should stop watching Storm Chasers while soldering!

It took a minor hack on the DS2438 library I’ve been using to add a function to read data from the ADC without converting to a humidity value with temperature compensation to get working, but seems to be pretty good so far; my reading are within 0.02 inHg from what I’ve seen so far, though only testing over the full 3-4″ range will evaluate the precision correctly.  I’ve updated the patch on my Google Code repository to reflect the new readAD() function.

There are some minor bugs I’ve noticed, namely the DS1820 temperature sensors can report invalid readings when the Arduino is powered on.  It only seems to happen when it’s reset, a cold power-on rarely causes it and usually an additional reset or two fixes the issue.  The DallasTemperature library is coded to use parasitic power for the sensors and it’s possible I may have wired them to use +5V; I’m thinking this could possibly cause issues but will need to check.  The lightning counter code currently calculates the derivative between readings and only uploads the differences rather then the absolute value, occasionally this will give an incorrect reading as well which results in a large negative value being transmitted.  These two issues can skew the scaling on the graphs which can be annoying.

I’m also looking into better integration with Pachube, I’ve seen a nice example which use EEML XML format with the API v2 which allows for better flexibility, it could handle the invalid sensor readings better by omitting them from the feed update.  I also came across a Perl library which I may use to write a small command-line client for updating or manually changing sensor readings…  perhaps a simple hack to fix the min/max graph scaling problem.

Although I consider this project something of a success I will eventually want to develop a new version from the ground up using sensors similar to the weather sensors SparkFun carries…  it will probably be similar to the USB weather board they built (using different I2C/SPI sensors) but I want to integrate a LCD and ethernet directly on the board.  Plus it will give me an excuse to finally make a custom PCB! 🙂

Images of the build:

Pachube datastream:

Dark-detecting Halloween Pumpkin

So I came across a project some time ago by one of my favorite electronics groups, Evil Mad Scientist Labs, where they put together a simple circuit for detecting the dark and illuminating LED “pumpkin eyes.”  It basically consists of a phototransistor, 2 NPN transistors, 2 LEDs, some resistors and a battery.  I’ve been meaning to do something like this for a Halloween project one of these years and finally decided to built it.  However since I originally read the article I stumbled upon a hack done on this by Jeroen at where he made a minimalistic version of the circuit  (based off a Flickering LED Instructable) where all of the components where removed except for the LEDs and microcontroller.

I love the fact this can be optimized so much and the results still work rather well.  Jeroen explains that it’s not commonly known that LED’s can be used as light sensors as well as their typical use so that’s what is done here…  one of the LED’s samples the light intensity and if it’s below a threshold the blinking starts.  When not running the microcontroller (ATtiny13V here) is put to sleep and actually does a very good job of conserving power (few uA in standby).  I won’t go into the details here, you can view the posting at to get more information.

I think my version has a small bug or something in the implementation as it’s not as responsive as his and the LEDs only flicker for a few seconds, then stop for a few, then resume and so on.  Probably something I overlooked in the code somewhere, but you get the idea seeing the video:

Some pictures of my build:

More available on my Flickr MAKE set.

Arduino OneWire/Pachube project uploaded

After uploading my radarwatch project to Google Code yesterday, I went ahead and created another one for my Arduino projects. Right now it consists of a basic weather station which reads in data from OneWire sensors and upload it in real-time to a live Pachube feed via an Ethernet shield. The sensors I’m using are from Hobby Boards and include temperature, humidity, solar, and lightning detection. I’d like to thank the developer ‘gfb’ for writing and open-sourcing an OneWire Arduino library for additional 1-wire IC’s beyond the standard DS1820 temperature sensor covered in the Arduino DallasTemperature library.

The code is pretty basic now and probably somewhat buggy.  I need to patch 2 of the OneWire libraries, one to fix a bug reading in DS2423 counters (lightning detection) and another patch to add functionality to read current from the DS2438 smart battery monitor (used as a solar sensor here).

Sensor data is displayed on a small 16×2 LCD display, echoed out to the USB serial port, uploaded to Pachube in real-time and also available via a webserver running on the Ethernet shield.  I’m not a huge fan of the sample tutorial code I used from Pachube, it seems kinda hackish.  I’ve seen better examples on there built around a nice library but unfortunately they all seem to need Processing, which I didn’t feel like using here.  Maybe if I get time I’ll put together a nice Pachube stand-alone Arduino library.

There are a few kinks to work out still and further development, such as getting the solar sensor close to the window to track daylight hours without compromising the accuracy of the thermal readings (old apartment, window insulation not so good).  I’ll try to post some pictures of the setup and maybe a schematic up soon.

If you’re curious, you can view the feed here:

Wiki entry :

Sample graphs: (these are real-time live data)

  • Temperature
  • Humidity
  • Lightning strikes (frequently produces false positives)

Sample images of the hardware:

radarwatch uploaded to Google Code!

So I finally got around to uploading my WSR-88D NEXRAD radar analysis project source to a new Google Code project. I’ve been wanting to do this for some time now and I’m sorry to admit that there is no documentation as-of-yet. Just wanted to get something uploaded and it’s probable that I will tinker with the repository further. There are also a fairly large amount of uncommitted changes which I need to work on before adding. Hopefully I will have some wiki docs up in the near future. I also plan to great another Google Code project to upload my Arduino 1-wire weather station code to as well.

I’d just like to extend a thank you to Nathan @ IBM for publishing a tutorial on writing your own weather alerts with GD which got me started on this project in the first place.  I don’t plan on ever making this production grade but hope to implement some features similar to professional projects such as NOAA’s CODE and NCAR’s TITAN.

*Update*: Some preliminary wiki docs have been added.

Ktux Multitasking & Update

So it’s been some time since I posted, unfortunately life has kept me on the go these past several weeks and I’ve had limited time at my desk.  However, thankfully I was able to work on Ktux from my netbook and now I successfully have multitasking working!  I attempted several previous implementations of a scheduler without success, this time I re-wrote mostly everything from scratch, minus a few lines of the timer ISR assembly code.  Currently, the scheduler supports thread creation/destruction and basic priority adjustment with nice() (just adjusts the process quantum timeslice).  Oh, and a simple yield() was implemented as well.  As of the moment, I have a pointer array to keep track of the threads, but in the process of replacing with linked lists (same implementation Linux kernel uses).

The screenshot below demonstrates 4 threads of execution, 3 of them increment counters (with various priorities) and the final one just displays the values on the screen, then yields():

I finally invented something that works!!

Now, of course with with multitasking support comes a whole new array of synchronization issues.  I have intentionally followed the KISS principle thus far when developing the kernel framework, knowing that any substantial code would most likely need to be re-written with critical sections, locks, etc.  Due to the complexities and time involved, I am fairly certain I will not aim for full SMP support in Ktux.  A few basic atomic operations have been written, but for the moment I think I will just use something similar to the big kernel lock in Linux; performance is not my main concern here, getting something out the door that actually runs is.

Not sure where I will go from here, I know the virtual memory needs a *LOT* of work, it’s pretty ugly right now.  From there, I think user-level thread support will be a logical next step.

Update & Ktux upload

So life has been quite hectic lately with the holidays and it’s time to get back to writing.  I still need to upload the final piece of my service account script which basically consists of a simple Nautilus script to invoke it.  This will be coming shortly, hopefully later this week.

As for my OS Ktux, I am uploading a tarfile with the latest code for anyone whom may care to peruse the source.  Unfortunately it’s been longer then I would have liked since this has been worked on.  However with my new laptop I will be re-creating my development environment and should have more information available soon.