Project Diary 2004

This is just an informal way to keep the team appraised of your projects and progress. We're growing, we're distributed, and we're collaborating. So, tell us what you're working on, finished, or starting!

This is an archive of the diary from the year 2004. The current ProjectDiary has links to newer items and such.

• BobKrzaczek - 01 Dec 2004
  • The rest of the ADP came over to Windows under our creo build system in about a half hour. Feeling good about that.
  • Everything's coming together nicely. The Windows HINSTANCE hack is, I think, fully addressed. There's no reason we can't start an Imagine session now. There's only a half hour before the weekly status meeting, but I should have a test georect server running in an hour or two. Definitely by COB today, barring any major interruptions. More status here later.
  • Working towards not needing the special libraries from Leica; I think I've learned enough about the Windows loader that I can work with the non-debug versions they already supply. This is good, since they can't deliver the libraries we asked for, it ought to get us around that need. This final question gets answered today, at the same time I get a georect process running.
  • "barring any major interruptions". I should know when to keep my mouth shut. Well, I'm not going home 'til I get somewhere.
  • I have a generic framework for ADP data processors now. Geo-rectification is the first use of it. It's a raw template that we'll polish up later to make writing ADP data processors super easy.
  • I have successfully linked code that attempts to start up an Imagine session via Leica's LggmImagineSession^? code. It runs, but throws an exception; this isn't too much of a surprise, I expected something like this. The Leica code wants the UTM zone, for use in its projections. Fine; I know that, most of New York is in UTM zone 18T. However, the API wants an integer, and a value of 18 throws an exception (as it should). So, how do I encode the UTM for Leica?
  • Looks like I'm at the point where I need to sit down with Dan. He can show me the code for calling geo-rectification in his project, and I'll transliterate it into the ADP. But, the important part is that I'm linking against the Leica software!

• BobKrzaczek - 30 Nov 2004
  • Port of ADP to MSVC 7.1 underway. I can't create debug libraries, unfortunately, but plain jane executables are working (so far).
  • Bug: For some reason, String_Base_Const::npos isn't in ACE.lib. Temporary workaround: added it to the ADP's aid library until we can get this fixed in ACE. Led to a successful link and execution of the controller on vulcan. This validates our Windows port!
  • It's 4:45am; I'm knocking off for a while; the rest of the ADP won't take more than an hour to port in the morning (now that the hard pieces are done). Then we try to round up all the little solutions to the problems with the Leica code into one executable and see if we can convince it to link. This stuff works fine separately, but in concert they failed. Now that I've updated creo to compiler flags from MSVC 7.1's ACE and TAO solution file, these problems might go away.

• BobKrzaczek - 29 Nov 2004
  • Major checkin to CVS of lots of code that I've been working on for the past few weeks. Seemed like a good time; things were getting old, and ScottLawrence needs an up-to-date IDL to work against for his new GUI.
  • Camera simulators based on the Warsaw flight data (from JasonFaulring) are up and being left on 24/7. Presently, I'm just bothering with the IR data, I'll bring up VNIR data later. Leaving these running and available all the time helps me shake out any memory leaks or other long term stability issues, and helps ScottLawrence and BillHoagland by having something for them to test against. The camera simulators are
    • /wasp/lwir
    • /wasp/mwir
    • /wasp/swir
  • Removed test image files from CVS, because the Terrapix images are too big and crash cvs up when it gets to the camerasim directory. Moved them to /lias/wasp2/wasp_collects2/oldADPtestset instead.

• BobKrzaczek - 14 Nov 2004
  • Finished moving the system to the new data format proposed back on the 5th.
  • Someone filled up the LIAS disks, though, so I can't do source control at all.
  • Failed to port Jerry's Java client over to the new data format. There's a subtle bug in our new data format with respect to Java. It's not hard to fix, but it's 7:30pm on Sunday, and I've got to work on SOFIA for the next couple days.
    • The bug, simply, is that a CORBA Union with multiple cases sharing the same clause doesn't maintain the discriminator. A discriminator will be set, yes, but there's no guarantee it's the same as the one used by the creator, only that it's "one of" the value used by the branch. Trawling through the JAVA/IDL mail archive on the net, by the people who actually wrote the JAVA IDL mapping standard, seems to imply this is the case. I won't write exactly what I think of this behaviour, other than to note it's unlike every other language mapping for CORBA and is completely insane. Freakin' language lawyers; they found something that didn't specify "must", and then proceded to read the surrounding text as creatively as they could.
    • Our fix is to include the image band inside the ImgData type, and include the event group inside the ApplEvent type. Yes, it's redundant, this is exactly what the union discriminator is for. But, doing this will insulate us from this and similar java stupidity.
  • On the other hand, controller, camerasim, norm, and watch/imgmon are all ported.
  • Creo now generates Java jarfiles for the IDL automatically. I have verified that this is working with the Java GUI (above).
  • Still thinking about the clock skew problem. I think I have a design in my mind, I need to sketch it out later this week when I can come back to work on WASP.

• BobKrzaczek - 10 Nov 2004
  • Successfully incorporated JasonFaulring's new test dataset for us into the camera simulator.
    • Wow, MWIR data is bizarre. Every other pixel is a stone cold zero, and the dynamic range of data is something like [0,240] (in a 14 or 16 bit system). But, I'm assured it's working as designed, so...
    • The data I got from DanielFava a while back and the data I just got from JasonFaulring have different byte ordering in the image files. This, you can imagine, is not pretty. Apparently I can deduce it from the image file name: .bin files are big endian, and .img are little endian. Rather than hardcode filenames into the ADP, the WASP camera simulator now accepts a keyword endian in its configuration file, taking an argument of big or little. Seems to address the issue.

• BobKrzaczek - 05 Nov 2004
  • After bouncing the idea off HarveyRhody and ScottLawrence, I've gone ahead and generalized the idea of a WASP image to two sequences: One, a sequence of octets (bytes) holding the image data, and the other, a sequence of dimensions. This lets us manage 1, 2, and 3 (or more) dimensional data fairly easily, and represent Indigo, Terrapix, and scaled Terrapix data in the same structure. This means a little more work in the client software, but pays off in the uniformity with which we deal with images.
  • Currently adding Java support and formal Windows support to the creo build system.
  • Leica will have status for us on Wednesday (10 Nov) regarding static library builds, migration to MSVC 7.1.
  • I agreed to try and hunt down the Windows Hinst issue that keeps us from starting an Imagine session in our programs; no one is shedding much light on this issue, unfortunately.

• BobKrzaczek - 02 Nov 2004
  • Hmm... While finishing up some demonstration stuff for tomorrow's WASP meeting, I managed to get the LWIR camera simulator to pause long enough to miss a firing command from the WASP controller. While it picked the firing commands back up and fell back into sync, as we designed, it wound up generating data with its frames "off by one" compared to the SWIR and MWIR cameras. Fix: Need to move the generation of frame numbers out of the cameras and into the controller, where it's global to the system, anwyay.
  • I hope you all voted.
  • (8:45pm) Yeah, it's official... there's too many opportunities for clock skew in the relationship between the controllers and the camera simulators (and, therefore, tne cameras). So, I'm going to move the sequence numbers into the controller. I'm also going to change the timing for the cameras; aesthetically, having a thread for the different cameras with their own timers is pleasing, but the reality of hardware requirements says otherwise. We'll shift to a single timer and "fire every N" counters for the cameras. That'll force everything to stay in sync.
  • Found and killed the bug with the VNIR camera simulator. The problem was that the simulators create data frames on the stack before distributing them to clients. An Indigo frame is 638 kB; that's no problem. A Terrapix frame, though, is 12 MB on the stack; that's kinda huge for a single data object. Needless to say, the camera simulator chokes. The fix was to make this a static structure that's continually reused; this is good, but it has the caveat that no camera simulator process can serve more than a single VNIR. No big deal. I think.

• HarveyRhody - 23 Oct 2004
  • Radial distortion correction tables have been produced for the SWIR, MWIR and LWIR cameras. They are tables that look like SWT.dat, SWC.dat; MWT.dat, MWC.dat and LWT.dat, LWC.dat on Vulcan C:/harvey/align. An example of how to do a distortion correction is in SWcorrect.pro, LWcorrect.pro and MWcorrect.pro. The corrected images look ok. I will be working on the construction of two more sets of tables, needed to register the LW and MW images to the SW image. Each table production run on Vulcan takes about 5.5 hours. I need to learn more math tricks to reduce this, but it is at least workable for now. We will only need to run these when something about the cameras changes.
  • It looks like correction of one image takes about 0.062 seconds on Vulcan using IDL. We need a C program to do the correction calculation.
  • To run the correction program go to the vulcan C:/harvey/align directory, double click on SWcorrect.pro and type @SWcorrect in the IDL command line. Similar for LW and MW.

• BobKrzaczek - 20 Oct 2004
  • While I'm on hiatus for a few days to work on SOFIA tasks, Leica has agreed to investigate two issues for us.
    1. Their modules are presently built under MSVC 7.0. I've asked them to either switch to 6.0 or 7.1; 7.0 has a lot of bugs, and isn't even supported by some other software packages we rely on (like ACE/TAO). Presently, their software won't build under 7.1, due to some older-style constructs in their code. They're going to look at updating their code or dropping to 6.0 or something for us...
    2. They will also look at packaging their release as a .LIB instead of a bunch of sources that compile to .DLL files. This matches the way we work with other libraries in the system.

• BobKrzaczek - 19 Oct 2004
  • Yeah! (2:17am) With some help and clues provided by some MSVC-using friends and Google, I'm building the WASP software, on vulcan, using MSVC. This is significant, as it clears the way for me to start building and linking against the Leica software. I should also note, these builds are successful using MSVC and GNU make, not the IDE. This means we get to keep our goal of "one set of source code, one set of build tools". The alternative would have been different build procedures for the Unix and Windows environments, which would have inevitably led to things falling out of sync with each other. This is a huge relief, I must say!

• BobKrzaczek - 18 Oct 2004
  • Got some good clues and hints from some friends that are professional Windows developers. To be honest, our problem stumped them as well, but they offered enough clues that I think we'll be out of the woods very soon.
  • Rebuilding ACE/TAO on vulcan with MSVC 6.0, based on aforementioned comments.
  • Partial Success (6:30pm) The WASP ADP compiles cleanly on vulcan. Now I'm working on getting these objects to link together into an executable cleanly.

• BobKrzaczek - 17 Oct 2004
  • WASP Controller, Camera Simulator, and "watch" (debug tool) modules are all working together. Yeah! Just need ScottLawrence to update his image client to the new IDL.
  • Port to MSVC is underway. Working with explicit makefiles in the meantime to save time, I'll port things back to creo later. So far, I have what I think is the IDL compiled, and the aid library beginning to compile, but it looks like I have to stop and find out why the ACE/TAO compilation didn't succeed back when I thought it did (under MSVC). I could really use some help from someone that understands development under Windows using Visual Studio .NET 2003; I know I'm missing some important commandline options for the compiler and linker, but they've moved since the old studio 5/6 and I can't find all of them anymore.
  • Outstanding bugs:
    1. Camera simulator breaks when working with Terrapix data?
    2. Controller object starts, runs, controls, but can't shutdown. Lack of proper thread join/cancel operation.
    3. Lack of synchronization in Controller between camera service threads lets them drift.
       • Option 1: remove separate threads for SWIR, MWIR, and LWIR, instead a single thread that fires all three together. Pro: Easier. Con: Leaves Terrapix firing with its own clock skew.
       • Option 2: Compute synchronization intervals between Indigo cameras. Also, could compute a synch. intervals between Indigo and Terrapix cameras (least common multiple) and bring them in sync that way. Pro: Brings all four cameras into sync. Con: Harder.
  • I'm in serious trouble with respect to SOFIA. WASP is taking 100% of my time.

• BobKrzaczek - 16 Oct 2004
  • Upgraded to Sun Forte 8, C++ 5.5, on chapman.
  • Upgraded to ACE 5.4.1 and TAO 1.4.1. Use an ACE_ROOT of /lias/archs/solaris/acetao541 if you want to use it. This is a debug build, I'll build an optimized one later.
  • Upgraded creo to work with the Forte and ACE/TAO upgrades.
  • Fixed the WASP controller object, finally, after days of mysterious crashes. It turned out to be an issue with an object var not always taking ownership of an object ptr, ultimately screwing up our memory management.
  • The controller reliably manages the LWIR, MWIR, and SWIR camera simulators. Now that we've got a reliable controller, the VNIR camera simulator shows us a bug. Not a huge issue at the moment, since our focus right now is IR data and the Leica software modules.
  • Next step: porting back to Windows/MSVC.

• BobKrzaczek - 13 Oct 2004
  • Finished the rewrite of the camera simulator; it now has to be triggered by an external object (the so-called controller), supports multiple cameras in a single object, supports the new IDL, and generates exterior orientation parameters when it can (that is, when they are defined). Woo!
  • The WASP ADP source repository can be browsed here.

• BobKrzaczek - 12 Oct 2004
  • Completed porting over the support libraries to the new IDL.

• BobKrzaczek - 11 Oct 2004
  • Completed the new simplification of the WASP IDL. Updated the UML as well.
  • Ran into another problem linking against the Leica software. The Leica modules are compiled with MSVC; even if I recompile them with g++, they still can't interface properly with the Imagine DLLs. The solution is going to take an extra day or two, but it'll mean recompiling all of ACE and TAO with MSVC, and then porting the ADP over to MSVC. (Yes, I mean "porting", g++ and MSVC are not on the same page regarding C++ standards. Example? friend declarations)
    • Aside: here's the basic problem... even though Microsoft, Borland, and Intel all agreed on a common ABI for their C++ code, thus enabling them to get around the standard C++ symbol mangling issues, the mangling implemented by the GNU people is not compatible.

• ScottLawrence - 11 Oct 2004
  • From a request by Bob, I left one ADP camera simulator and one image viewer running through the weekend. After 100,000 images, it was still running with no memory leaks. Yay!

• BobKrzaczek - 10 Oct 2004
  • Completed the rewrite of the WASP ADP interface definitions.

• BobKrzaczek - 09 Oct 2004
  • Fixed some bugs in the waspnw documentation class that prevented us from generating DVI files. Now, we can not only generate DVI, but tex4ht works on the Wasp LP documentation as well.

• ScottLawrence - 07 Oct 2004
  • I just finished the first version of the image viewer client, which I had discussed at the last meeting. This is a java application which connects up to bob's camera simulator and displays the images, as they are captured. ... So, we we can have java applications (the future GUI) talking with c++ objects (the rest of the system) and display imagery no problem.

• BobKrzaczek - 07 Oct 2004
  • Finished the rewrite of the build tool creo based on our experience with the previous one. This one seems pretty robust. Supports Solaris/Forte and mingw32/gcc.
  • Brought the developer documentation back up to date.
  • I'm starting to put our documentation online. You can download and read the developer's introduction to the WASP ADP, as well as the entire source tree, over in the SystemDesign^? topic.