Note Yep, this topic is officially out of date. The interfaces and design, in particular, are migrating to CVS and noweb (literate programming). I still want to hear your comments if you have them; chances are, the changes I'm making and the changes you suggest aren't related.

Note This topic is "semi-closed"; I'll update it to reflect bugs and changes in the overall system design, but as we enter coding and implementation, this topic may fall out of step with the actual software being implemented. This topic was intended to develop the initial concepts about the system in an open environment, which it succeeded in; we aren't yet at a point where a wiki can be used for literate programming. (hint, hint!)

Table of Contents

Introduction

This paper presents the design for the WASP in-flight software system. It should present not only the "what" of the design, but also the "why"; references to other topics will be made freely. The authoritative version of this document shall always reside on the WASP wiki. If you are a registered user on this wiki, you are free to edit and refine this design yourself; alternatively, you can make a short comment at the end of this doc.

Overall

The overall goal of this system is modularity. As the system is implemented by no single group, we require the ability to swap new and updated versions of software in and out easily. It is likely that multiple instances of this software will be run side-by-side, in order to evaluate variations both in algorithms provided by other groups, as well as in our own control framework. The WASP team has the responsibility for producing the overall software system, into which deliverables from other groups will be combined.

Inputs

• WASP Team
  • design and implementation of overall software system
• Leica
  • design and impl of geo-rectification of images from the WASP sensors
  • design and impl of co-registration of imagery
  • design and impl of image transformations supporting in-flight display
  • design and impl of image transformations supporting post-flight high-quality mosaicing
  • Note: it is expected that RIT will "wrap" these four modules for inclusion in our system
• DIRS Team
  • designa and impl of fire detection algorithm
• Pixel Physics
  • Terrapix software control and processing
• Mike Richardson
  • Scenarios for WASP operation, both in-flight and on the ground (at the fire camp).

Constraints

At first, WASP data acquisition will be hosted on a single computer running Windows Server 2003, supporting 2 processors. We have no choice but to run Windows, despite its lack of real-time support, as that is the only operating system for which Pixel Physics will deliver the software supporting the Terrapix camera. Our software, then, cannot run near the edge of performance for this system; if it did, the lack of control over scheduling would introduce delays and we would slowly, but surely, fall behind as the flight progreses. Up to four processors may be supported by the current WASP computer.

Disk space should not be a problem; a large RAID will be supported in the plane as well. The size of this RAID was jointly specified by some members of the WASP team and members of the RIT Computer Engineering department, and will provide adequate space to record all raw and processed data generated during a WASP flight.

Despite the RAID, the WASP software should not rely on disk-based files for any inter-process communication in the system. As much as possible, data should be managed in-memory, or copied between buffers provided by software objects. Relying on writes and reads through the Windows I/O layer to an external disk will introduce far too much overhead to sustain the necessary system performance required by all four sensors.

The system has two modes of operation.

• Day flights in which the VNIR, SWIR, MWIR, and LWIR sensors are engaged.
• Night flights in which the VNIR sensor is not used.

The system collects data from its cameras relatively fast.

• Initial data rates
  • VNIR: 1 frame every 4 seconds
  • SWIR, MWIR, LWIR: 1 frame from all three every 2 seconds
• Long term goals
  • VNIR: 1 frame every 2 seconds
  • SWIR, MWIR, LWIR: 2 frames from all three every second

Organization

The WASP acquisition software shall be organized as a collection of distributed objects communicating over CORBA. The WASP system will be self-organizing, regardless of the number of computers WASP is deployed on, or the number of processors available in each computer. Simply starting the WASP system server processes should be enough to yield a running system; no special human input should be necessary. Not only it is our goal to make this as "turn-key" as possible, but scalability should be easy to achieve as well.

All objects shall be self-configuring.

• An object may provide a rich interface for controlling its behaviour; however, it may not require its use. Put another way: while the user can make changes and adjustments as necessary to any given object in the system, the objects must not require such input in order to make themselves ready for processing WASP data.
• Without any refinement from the operator, the object should select conservative settings. The meaning of "conservative" in this case is left up to the object's implementor, but should reflect the object's best chances for providing useful data reduction under a variety of environments..

The CORBA CosNaming service shall be used in order for objects to locate and post their runtime locations. A single well known NamingService will be made available to all WASP objects.

• It is recognized that this service is an additional program to be run in the WASP processing system, and must be run prior to all other WASP objects.
• Once the system is running, however, this naming service will probably fall idle and present no real load to the rest of the system.

All interfaces (e.g., sensor frame formats, control interfaces) will be specified using CORBA IDL.

A simple "push" model will drive the flow of data through the system.

• We recognized that some of the data moving through the system will be large and fast.
  • 32MB from the Terrapix every 2-4 seconds
  • 640kB from each of three Indigo sensors, up to twice per second
• For that reason, we initially specified a PushPull model, thinking that it would be best for the models to only pull the data they need at any time.
• However, this turns out to be an unnecessary complexity: each image is unique, and must be transmitted all the time. The extra refinement in a PushPull model goes to waste, since there is never a situation where a processing module in our system doesn't want data from a previous module.

CORBA Naming Service

There will be a single CORBA Naming Service provided for WASP. Since it's likely that multiple deployments of the WASP acquisition software will run simultaneously on the ground, supporting test and development efforts, each instance of the WASP software will use a common naming context. This will be located in the CosNaming initial naming context.

The default name of this context shall be WASP-Acq. When the WASP system is started, the operator may specify an alternative name to support private development or test efforts. All other objects, parameters, and information shared in the CosNaming service shall be located in this context.

For example, consider this listing from a CosNaming service. In it, we see that there are two WASP acquisition systems running simultaneously. One appears to be the "official" system, currently running in a night flight configuration. The other appears to be an engineer's private test environment, supporting his investigation of different operator console displays (snail trail and waterfall models). These are not authortative example, but merely provided to show the overall structure that the CosNaming service will be used to support.

|-- Wasp-Acq
|   |-- ImgRegis
|   |-- LWIR-geo
|   |-- LWIR-hw
|   |-- MWIR-geo
|   |-- MWIR-hw
|   |-- OperatorConsole
|   |-- SWIR-geo
|   |-- SWIR-hw
|   `-- ThumbMaker
|
`-- bob-test
    |-- ImgRegis
    |-- MWIR-geo
    |-- MWIR-hw
    |-- OperatorConsole
    |-- SnailSvr
    |-- ThumbMaker
    `-- WaterfallSvr

High Level Objects

So, what are the objects in our system, exactly? We aren't going to specify every single object in our system; many of those decisions will be left to the engineers performing the system implementation. However, a high level understanding of certain key objects is required.

Loosely, the system can be broken into three categories.

1. Multiple sensor-specific object chains
2. A single multi-band object chain
3. A user-interface and control object

Interfaces

Everything we do will be presented within a namespace (IDL "module") named WASP. Previously, we also enforced an object prefix of cis.rit.edu but we've since abandoned that: the TAO IDL compiler got confused with the nesting of multiple idl files all using the same prefix.

Base Object

The BaseObject interface is not intended to be used directly; rather, it will be inherited by just about every other object in the system. This provides a standard set of tools for querying the health and status of objects in the WASP system. The code in this subsection should be collected into base.idl.

module WASP {
    interface BaseObject {

An object moves through three states as it comes up. The last state, NotReady, is only moved into when something goes wrong. It's undefined whether or not reset() "must" or "should" move the object out of NotReady and back to Initializing. I'm leaning towards "should", as there could be errors at startup time that prevent an object from ever becoming "ready".

        enum ObjState { Initializing, LookingForPeer, Running, NotReady };

When requesting the status of an object, we should return not only the object state but also some text intended for an end user further describing the state. Thus, if we're NotReady, the extra text should indicate why.

        struct Status {
            ObjState state;     // the state of the object
            string extra;       // any extra text, maybe useful to humans
        };

One should be able to call getStatus() whenever they like (within reason).

        Status getStatus();

shutdown() is a "once only" call; it will instruct the object servers to exit. It's a harsh call, generally you would only run it on a quiescent system.

        boolean shutdown();

reset() tries to put the object back into a known state; most initializations are lost, and the object should shift back to the first Initializing state. Internal data structures should be freed or cleaned up as much as possible.

        boolean reset();
    };
};

Camera

Camera geometries, internal and external. This maps, more or less, to Leica's IoParams and EoParams structures. It isn't 100% clear how we'll be using this yet, but I'm certain we'll need it (probably around initialization time). This interface is pretty simple, there isn't much to say about it. The code in this subsection should be collected into camera.idl.

#include "base.idl"

module WASP {
     interface Camera {

          struct InteriorOrientation {
               double ccdSize;          // millimeters
               double f;                // camera's focal length in millimeters
               double p0x, p0y;         // x,y of camera's principal point
               double k0, k1, k2;       // radial distortion coefficients
          };

          InteriorOrientation getInteriorOrientation();

          struct ExteriorOrientation {
               double o, k, p;          // omega, kappa, phi rotations
                                        // about the x, z, y axes
               double x, y, z;          // sensor position in 3D space
          };

          ExteriorOrientation getExteriorOrientation();
     };
};

Indigo

An "interesting" aspect of CORBA is that there's no formally defined mechanism for one object to "know" (get the reference to) another object that contacts it. So, rather than traditional push-pull interface (i.e., one that "pushes" a notification that results in a "pull" of the data), we'll define something only slightly different...

This is really a "pull" approach to data movement. However, rather than the incessant polling that we'd normally see in a system that does not have "on demand" data generation, we ensure the client sits idle until it is instructed to get its data. This call typically comes from a producer, but includes the object reference from which to pull the data; this has some interesting flexibility we probably won't need at first (such as: the object generating data and the object monitoring the data needn't be the same). The code in this subsection should be collected into indigo.idl.

#include "camera.idl"

module WASP {
     module Indigo {

A short list of the different bands collected by this class of instrument.

          enum CameraBand { LWIR, MWIR, SWIR };

Indigo images...

          const unsigned short WID = 640;
          const unsigned short HGT = 512;
          typedef unsigned short ImgData[ WID ][ HGT ];

Indigo Image frames; we'll probably accumulate timestamps, sequence numbers, and maybe even some IMU/GPS information in here over time.

          struct Image {
               CameraBand band;
               ImgData img;
          };

Indigo Image Providers: Another object calls our getImage() function in order to obtain a frame of image data. It supplies an token, telling us which frame of image data to return. There's no guarantees about the lifetime of any particular frame of image data; in fact, it's quite likely that after an image frame has been retrieved, it'll no longer be available.

          interface ImageProvider {
               Image getImage( in unsigned long token );
          };

Indigo Image Consumers: An image consumer is notified that a new frame of image data is available from some other object via imageReady(). A token is supplied which tells the image provider which image frame is requested; the consumer doesn't need to know what the token really represents, but it'll probably just be an image sequence number. The return value from imageReady() indicates whether the consumer accepts this notification; at present, this really isn't used for anything, but in a robust industrial-strength system, we'd pay attention to it.

          interface ImageConsumer {
               boolean imageReady( in ImageProvider obj,
                                   in unsigned long token );
          };

This defines the Indigo camera itself, which provides camera orientation data as well as frames of Indigo data.

          interface IndigoCamera : WASP::Camera, ImageProvider, BaseObject { };

This defines the Geo-rectification server, which accepts frames of Indigo image data from the camera, straightens them out, and then passes them onto the band-to-band registration object. As such, this object is both a consumer and a producer of Indigo Image data.

          interface GeoRect : ImageProvider, ImageConsumer, BaseObject { };
     };
};

Terrapix

The Terrapix module is exactly like the Indigo one, modulo some changes to the bands and image sizes. For the moment, we're ignoring differences in the underlying meaning of the image data (for example, the Indigo emits single wavelength (monochromatic) image data, while the Terrapix generates images with three wavelengths (visible red, green, and blue) interleaved in a Bayer pattern). Therefore, it's easier to generate the Terrapix module from the Indigo module, rather than manage redundant information in two places. The code in this subsection will generate terrapix.idl.

terrapix.idl:   indigo.idl
        expand $^ \
        |sed -e 's/Indigo/Terrapix/g' \
             -e 's/WID *= *[0-9]*/WID = 4096/' \
             -e 's/HGT *= *[0-9]*/HGT = 4096/' \
             -e 's/enum CameraBand {.*};/enum CameraBand { VNIR };/' \
             >$@

One likely evolution of the Terrapix module will be

• Reducing the data from Bayer-interleaved pixels to true RGB tuples, which leads to
• Changing the size of the image from 4,096×4,096 to 2,048×2,048 or even 1,024×1,024 (depending on how the Bayer pattern is processed).

Band-to-band Registration

There's just a single one of these in the system. It pulls data from Geo-Rectification objects (one per IR channel) and performs some sort of pixel or sub-pixel alignment and adjustment. The results are then passed to the fire detection object.

What the heck does this mean? Kidding aside, we don't know what it means to adjust the images; do we simply emit rotations and shifts, or should we expect the registration object to emit new images? It's not clear. Can we even reuse the Indigo:Image, or should we define a larger image to handle whatever shifts and rotations are necessary. If so, how much bigger? And then, won't we need to specify some sort of image mask so we know which pixels are real and which are transparent/ignored? The code in this subsection should be collected into bbreg.idl.

#include "indigo.idl"
module WASP {

The band-to-band registration object is, presently, only a data consumer until we define how it talks to the fire detection algorithm.

     interface BBRegister : Indigo::ImageConsumer { };
};

Console

Okay, this is going to be pretty sparse until I get a better understanding of what the GUI needs are from Jason. In the meantime, though... The code in this subsection should be collected into console.idl.

#include "base.idl"
#include "camera.idl"
#include "terrapix.idl"
#include "indigo.idl"

module WASP {

At present, all we know is that the console will consume both Indigo and Terrapix image frames... from where? Who knows. Also, we can't inherit from both Indigo::ImageConsumer and Terrapix::ImageConsumer; well, we could, but CORBA specifically disallows this because it's afraid of trying to map that to non-OO languages. C++ and Java would have no problems here at all. sigh. So, we'll have to craft our interface by hand, instead of by inheritance.

     interface Console {
          boolean indigoImageReady( in Indigo::ImageProvider obj,
                                    in unsigned long token );
          boolean terrapixImageReady( in Terrapix::ImageProvider obj,
                                      in unsigned long token );
     };
};

Fire Detection

There's just a single one of these in the system. It pulls data from the Band-to-Band Registration object, thinks really hard, and then tells something else if there's a fire in those images and where.

What the heck does this mean? We don't yet know what shape our data is going to have, or what we're going to notify, or how, or... The code in this subsection should be collected into fire.idl.

#include "indigo.idl"
#include "console.idl"

module WASP {

The fire detection object is, presently, only a data consumer (from the registration server) until we define how it talks to the console.

     interface FireDetection : Indigo::ImageConsumer, BaseObject { };
};

---

If you'd rather not edit this document, you can just leave a comment here instead.

* We should ensure we spend our effort on those things that are necessary for a successful demo/prototype, not polishing for a deployed system. We can tolerate more manual intervention than the eventual system. -- DonMcKeown - 16 Jun 2004 13:54:13

• There will be, like SOFIA, a pre-flight prep phase, and dead time in the air (on turns, pre-flight taxi, etc). We need to specify what happens during these steps; especially during the pre-flight. What cameras will be run? What are the flight plans? What is "loaded" into the "electronic footlocker"? Note parallel to SOFIA AOR. Need to keep this in mind to be specified later, as we grow the system control portion of the design. Some sort of overall flight description (like an ObsPlan), something that configures, or is a part of the configuration, of the objects in the system before take-off. -- HarveyRhody - 16 Jun 2004 13:48:37

• Nick Schad (Pixel Physics) will be writing a new Terrapix driver, an "ultimate" driver to support "all" platforms. HarveyRhody notes the risk of "vaporware", so we need to get a schedule or timeframes. They'll still release the information to us, regardless of the driver. The existing driver does work, delivered as a DLL, and is callable from a normal C++ object. Delivered as two pieces, a "low-level" that provides the GRGB Bayer data, and a "high-level" that provides RGB data. -- JasonFaulring - 16 Jun 2004 13:41:17

• Applanix data is an input to Geo-Rect services. -- BobKrzaczek - 14 Jun 2004 15:59:36
  • Captured. -- BobKrzaczek - 16 Jun 2004

• Split UI/Control module into two pieces: System controller that loops & fires cameras as spec'd before the flight, and UI that provides control (stop/start, system health, etc.) -- BobKrzaczek - 14 Jun 2004 15:59:08
  • Done. Not fully captured in the design picture above, but everyone's on the same page. -- BobKrzaczek - 16 Jun 2004

• Applanix (IMU, GPS, and event) data missing; need to add it. JasonFaulring notes it's similar to a camera, modulo being free-running, not "fired". Thinks there might be some utility to being start/stoppable ("would be nice"). -- BobKrzaczek - 14 Jun 2004 15:58:10
  • JasonFaulring will present the interface and use model for the Applaxnix on Wednesday (June 16). After this meeting, I'll update the design. -- BobKrzaczek - 16 Jun 2004

• Image Co-Reg server will need some sort of input (from UI?) so that the operator can mask out one or more sensor chains (e.g., SWIR is suddenly not working right, we don't want Image Co-Reg stuck waiting forever on data it can never get). -- ScottLawrence - 14 Jun 2004 15:44:45
  • Later discussion revealed that this is more of a pre-flight configuration (or pre-flight test), but sure, we can easily allow for this. -- BobKrzaczek - 16 Jun 2004

• HarveyRhody notes that PushPull was chosen not just for "picking and choosing" data, but also for some sort of process synchronization. Some way it addressed falling behind, or handling a trailing sensor chain in some way. -- BobKrzaczek - 14 Jun 2004 15:39:05
  • We'll spec PushPull interfaces, then. I still can't remember how I expected to handle falling behind with it, other than simply "losing" data (and I've tried to remember!), but it's not much more overhead/complexity for a potential big win down the road. -- BobKrzaczek - 16 Jun 2004

• Video rate data processing may not even be a part of this system, might very well be another application instead. -- HarveyRhody - 14 Jun 2004 15:28:39
  • Okay, nuking it from the goals. -- BobKrzaczek - 16 Jun 2004

• Jason says we'll have full support for both Indigo (maybe) and Terrapix (definitely) from Pixel Physics this week (6/14). Full = h/w interfaces, etc, "enough to write a driver" if we had to. -- BobKrzaczek - 14 Jun 2004 15:23:03
  • I think it's too late for us to consider writing our own drivers. If we had this months ago, when we were talking with EMRT and other folks about real time operating systems, we could have taken advantage of this. At this point, I think all we can do is use this valuable information in debugging and describing the software we get from Pixel Physics. -- BobKrzaczek - 16 Jun 2004

• Maybe the terrapix data is decoded from its Bayer pattern before it even hits our software? If not, we need to add another module to the VNIR chain. -- BobKrzaczek - 14 Jun 2004 14:33:34
  • This can be implemented with software from Pixel Physics, or via algorithms that Scott and Harvey were playing with. Either way, it must happen immediately after data acquisition, before Corba is involved at all. -- BobKrzaczek - 16 Jun 2004

• Terrapix frame definition is almost certainly wrong. Record best idea from Jerry and Harvey? 2k by 2k, interpolating the red and blue data? -- BobKrzaczek - 14 Jun 2004 14:31:38
  • Still want input from ScottLawrence, but dropping from 4k to 2k. -- BobKrzaczek - 16 Jun 2004

• Need to drop the IDL written during benchmarking in here, to give the interface development a starting point. -- BobKrzaczek - 13 Jun 2004 21:20:45
  • Done. -- BobKrzaczek - 16 Jun 2004

---