ZeroMQ libraries & stability / radarwatch update

I’ve been hacking with ZeroMQ for some time now…  reading through the excellent guide and working on the preliminary steps to incorporate it into the new version (migrated to GitHub from Google Code) of my Doppler radar analysis application.  Multi-threading support was on the list of original criterion and considering many of the other analysis features I’d like to eventually add, full distributed processing would be the best design.  I’m not an expert hand at SMP application design and although the implementation needs here would be fairly trivial, choosing a good library to handle the nasty bits would make things vastly easier.

So I tinkered with several of their samples designs, implemented a few of the patterns in my code and tested things out.  Initial results with just a simple multithreaded service seemed work OK at first glance but eventually dropped data caused issues so I decided to try something with reliability such as the paranoid pirate pattern (robust reliable queuing).  Thinking back, I should have realised something was up when it exhibited that issue on both inproc IPC and tcp/localhost.  No matter what I was trying to do porting examples into my code nothing remained stable; either it crashed immediately or processed a few messages then died.  C and C++ code showed similar results.  Running the examples on their own worked fine.  The front-end/client-side Python application worked fine.  I gave it a fresh shot with another example – the Majordomo pattern (service-oriented reliable queuing) again showed similar symptoms of instability.  Commenting out huge sections of my code thinking it may of had something to do with a broken pointer or some other oversight on my part had little to no results.  Eventually I started comparing the GCC invocations and noticed there were some minor differences.  After 5 minutes of testing the weeks of tinkering finally yielded some fruit… Google confirmed my suspicions: linking ZeroMQ with -pg for gprof support crashes the library.

It’s amazing how quick it can be to find what you’re looking for when you know exactly what you want.  At least now I can get back to implementing 0MQ in my code.

Quick removal of comments from C/C++ code

Working on another post – writing an optimization HOW-TO with valgrind (using my radarwatchd project as an example) and unfortunately realized the code I have in my Google code repository is fairly out-of-sync with the latest as I was working on some significant patches.  The changes incorporated numerous performance enhancements but there was a lot of old code I simply commented out temporarily while developing the new source.  Needing a quick way to diff the two without seeing any comments I stumbled across a solution with Perl.

Pretty simple: I have the current code in a subdirectory ./radarwatchd and the old version from the Google repo in ./googlecode/radarwatchd.  I made a copy of the sources as to not modify the original files, then ran a Perl construct to perform the regex.  A simple diff wrapped things up.

find {,googlecode/}radarwatchd/cached/ \( -name '*.cpp' -o -name '*.h' \) -exec cp -v '{}' '{}.nocomments' \;
find {,googlecode/}radarwatchd/cached/ -name '*.nocomments' -exec perl -0777 -pi -e 's{//.*?$|/\*.*?\*/}{ }smg' '{}' \;
diff -NBwar {,googlecode/}radarwatchd/cached/*.cpp.nocomments

The Perl regex is not crazy complex and there are probably some instances where it doesn’t work quite right but for a simple search this works well.  The only downside I had with this is that the comments are replaced by a single space, so you’ll need to run diff with -w to ignore whitespace.  With this particular invocation you can’t replace it with nothing as the -exec will see that first {} as the place to put the filename (rather then the last) and cause all sorts of problems.


radarwatch uploaded to Google Code!

So I finally got around to uploading my WSR-88D NEXRAD radar analysis project source to a new Google Code project. I’ve been wanting to do this for some time now and I’m sorry to admit that there is no documentation as-of-yet. Just wanted to get something uploaded and it’s probable that I will tinker with the repository further. There are also a fairly large amount of uncommitted changes which I need to work on before adding. Hopefully I will have some wiki docs up in the near future. I also plan to great another Google Code project to upload my Arduino 1-wire weather station code to as well.

I’d just like to extend a thank you to Nathan @ IBM for publishing a tutorial on writing your own weather alerts with GD which got me started on this project in the first place.  I don’t plan on ever making this production grade but hope to implement some features similar to professional projects such as NOAA’s CODE and NCAR’s TITAN.

*Update*: Some preliminary wiki docs have been added.