Approach

This set of user interfaces for the sensor array and image capture systems is intended to allow users to browse captured data and images, and control the operation of the systems. The long-range objective is to deliver real-time and historical data displayed in detailed geographic context, along with data from sensors as they are moved to various locations and includes innovative work in sensor and data visualization systems using Google Earth. Recent work on Emissary has focused on database construction to allow a common data format for front-end applications in addition to the construction of individual tools for project-specific deployments.  With further development, this framework and set of tools will be used to facilitate information use and sharing among researchers, educators, and the public.

Systems/Experiments

Emissary is a project that involves advanced data visualization and control of sensors and deployments. There are many parts to Emissary and work is progressing to unify the structure. Currently, Emissary is composed of tools that allow users to (1) deploy, troubleshoot, and control static and mobile sensor networks, (2) visualize and graph sensor data collected by those networks, (3) analyze image data collected by those networks, and (4) explore all these data sets in an integrated, visual, 3-D system.

(1) Tools for Deployment and Control of Static and Mobile Sensor Networks.

The Deployment Analysis System (DAS). DAS is a collection of tools accessible on the internet that helps users understand and deploy static sensor networks.

Figure 27. The status display shows the health of each individual mote in James Reserve CMS. All the motes are returning data within 0.1hour. The battery level of Mosscam 6 (mote ID 12) is suspect because it is outside the normal threshold. The health bar below shows the status within the last few weeks.

Currently a few applications are built using the DAS database, including topological maps that display system metrics including link quality, next hop neighbor, ingress/egress, and other metrics that give users an overall view of the sensor network system health. Other tools include easy to generate graphs that help users correlate different modalities of the system such as temperature vs. battery, and many others.

NIMS Control and Image Capture Interfaces. For mobile imagers and sensor platforms, physical control over position with simultaneous data collection is important.  The following are example tools built for such a purpose.

Figure 28. During the Rhododendron leaf size and phenology deployment of NIMS 1, daily images were collected remotely via the menu interface, moving the NIMS imager into location and capturing images. Feedback on position and a list of historical image captures was presented.

The Emissary control interface for NIMS RD consists of a web server backend located on the machine connected to the NIMS RD motors and a MATLAB front end. A user can enter measured values and updated the NIMS RD system to reflect these values. The user is also allowed to save the current configuration for reuse in the future. This interface consists of three main functions, control, data processing, and data visualization.

Figure 29. The control interface front end for the NIMS RD system. There are two methods of control: actual and relative. The upper cluster of buttons allows the user to move the NIMS RD node to relative positions such as 100mm down or 5000mm forward. The lower set of inputs allows the user to move the node to exact locations by specifying an x y coordinate to move the node to. The calibrate feature is a NIMS RD function that aligns the NIMS RD coordinate system to the current position.

Currently NIMS RD employs a standard laptop to control the motors so Apache is used as the web server for the cgi servlets that act as a backend to the Emissary control interface. These servlets also work as standalone web pages for any browser. In a deployment application, a user with any wireless enabled laptop or pda device can control NIMS RD using the pda’s built in browser.

A recent addition to the Emissary interface has been a set of diagnostics tools. These tools operate in the same way that the control functions do, cgi servlets that that interface with MATLAB or work as standalone web pages. Information such as dmesgs, network configuration messages, and process information can be obtained from the NIMS RD server laptop to provide status and troubleshooting help during a deployment. This information can also be archived for future comparison.

(2) Tools for Visualization and Graphing of Sensor Data

Data graphing and James Reserve image capture and control tools.  Two more flexible versions of Sean Askay's 'DMS GUI' were developed this past year. The first uses a third-party java applet (EasyCharts) to plot data, allowing the user more flexibility in zooming/panning through data and altering the details of graph formatting without reloading the page. The second enables users to graph sensor measurement streams stacked directly on top of image capture time series.

Figure 30. EasyCharts java applet - highly configurable in the web browser via Javascript. Allows attaching labels to data points, changing trace colors instantly, multiple y-axes, and many other features.

Figure 31. DMS GUI ver. 3 - Multiple plots of multiple measurement streams and multiple image capture series. By clicking on the zoom / pan links at the top, users can refocus the timeframe displayed. By clicking directly on the image, the user gets a full resolution version of the thumbnail; clicking on the day (or hour, week, month) listed below the thumbnail, the user gets more images from that time-period (down to a 5 or 15 minute resolution). Because only a few images can be displayed across the bottom, represetives spread throughout the time frame are chosen automatically. If there are more images available for the time frame (down to intervals of 5min, 15min, hour, day, week, month) they are represented by black vertical bars above the image thumbnails. The yellow vertical bars represent those thumbnails already displayed.

NIMS Systems Visualization. Data collected by the NIMS systems is currently obtained from separate tools. All NIMS RD deployments, regardless of which array of sensors is attached consists ofmaking repeated scans of a transect. The Emissary data visualization environment allows data from different sensors to be linked together based on the time axis.  These linked graphs iterate over each of these scans.  This allows a user to browse through different sensors data and observe the relationships between different sensor readings.

Figure 32. Multiple NIMS Sensors being displayed using plot types.

The data visualization functionality of the NIMS Emissary environment incorporates the built in features of MATLAB’s visualization environment as well as enhancements tailored towards the types of datasets that NIMS RD generates such as scaling and shading options. Additionally, the Emissary environment allows a user to export the graphical data into a series of images. This allows a user to customize the type of visualization used to represent a data set and then generate a series of images using this configuration.

In addition to data synchronization between Emissary and MATLAB, this interface provides export functionality for both Microsoft Excel and a more generic comma separated formatted. This allows a user to use whatever data processing tools he or she is comfortable with to do more a rigorous analysis of the data.

In order to allow users without MATLAB to use the various NIMS RD interfaces, windows binaries were compiled and install programs were created. All necessary MATLAB libraries are included. This allows almost any user easily install and use these interfaces.

(3) Tools to Analyze Image Data

The database has thousands of images taken from various experiments, however easily examining and analyzing those images requires new software tools.  Work on such tools has been on-going and three examples (Histogram, Tower cam Control, and Spectral Response) are explained below. 

Histogram Tool.  Each image is linked to its RGB histogram for simple analysis and a table containing its metadata.  The webpage interfaces to these image sets are shown on the following pages.  The current experiment involves inserting images into the database from the tower cameras twice a day.

Figure 33. The interface displays links to all images taken from the specified start/end date and time from the selected image set.  Image links are sorted currently by time.  A link to the image’s histogram is also provided. Images are displayed in the lower right.  The histograms are displayed in the upper right.

The image set stored in the database from a lab at UCLA during physiological investigations associated with the Mosscam project were also analyzed with this tool.

Tower Cam Control.  A tool for creating panoramas of the images and then letting a user select a sub-image and zoom in on objects of interest has allowed a greater functionality and use for investigations. Currently, there are 4 towers located at the James Reserve that take low resolution images of the surrounding area to monitor phenology.  Higher resolution images of specific trees and plants will be targeted this spring. 

Figure 34. Tower Camera Control with coordinates of the sub-sections that make up a panorama.

Spectral Response Tool.  Characterizing the camera’s spectral response is one step in the process of analyzing images taken under continually changing daylight illumination.  Cameras, regardless of their location or use can be calibrated to act as low-resolution spectroradiometers for the measurement of changes in quality of light. The associated metadata includes the wavelength of light, shutter speed, and iris opening.

Figure 35, Spectral Response Tool.  Although this image set resides in the same database, this set has a separate interface to allow searching based on the wavelength, shutter speed, and iris.

(4) An Integrated 3-D GIS Interface Tool to James Reserve systems: Google Earth.  As the scale and complexity of TEOS sensor networks increase, so does the need for an intuitive, integrated systems and data visualization interface. Existing web-based interfaces at the James Reserve allowed only for the basic plotting of sensor measurements and the viewing of image array sequences. Geographic select of sensors was only possible via a simple, flat jpeg map. Thus, visual analysis of data was limited to only a few sites at a time, and failed to paint the 'larger picture' of the micro-climate and biological processes made possible by sensor networks.

If placed into a geographic context, parallel measurement streams offer powerful insights into the dynamics of three-dimensional micro-climate variability. For example, point datasets can be statistically modeled and spatially interpolated to estimate climate conditions between actual sensor locations (Kriging models). Sequential animation of such models allows researchers to look at climate and biological changes in four dimensions, including time.

Google Earth, a widely available '3D world' application, offers such possibilities to the visualization of TEOS sensor network systems. Because this program is designed for tight web-integration, tying it to our existing web-based graphing tools and sensor database has been relatively straightforward. Thus far, we have had great success using Google Earth as a top-level visual interface for TEOS systems (Figures 8 – 15). Available online at http://dms.jamesreserve.edu

The Google Earth interface to TEOS at the James Reserve. Sensors are symbolized by 3D polygons generated dynamically based on GPS locations stored in the DMS database. Users can click on the polygon labels to get a description bubble containing sensor information, GPS coordinates, and webcams, if available

Figure 36. By clicking on Nestbox 47, a description bubble appears which contains the most recent nestbox webcam image, current sensor values and a tiny 'sparkline' graph giving users a quick-look at microclimate conditions over the last 24 hours. By clicking on the “[...]” next to the sparkine, Google Earth's web-broser window pane opens to reveal an 'EasyCharts' graph of temperature of the last 24 hours. As mentioned above, EasyGraphs is high configurable at run-time. Measurement lines can be represented as 3D ribbons (as here) or as simple lines with markers at each actual measurement.

Custom Google Earth elements are created using Google's geographic markup language (KML). As an XML derivative, KML is easily generated with PHP (Pre-hypertext Processor). This scripting language was already in use to access our MySQL 'DMS' database and to run our existing graphing tools. Using the free Google Earth client, we are able to extract a system-wide snapshot of our sensor systems from the database and present a 'current conditions' view of the James Reserve. From the extended descriptions of each sensor-site location, users can navigate through graphs of past measurements and view the corresponding image time-series from plant phenology and avian biology webcams.

Figure 37.  Current Nestbox and RoboCam webcams are easily accessible as thumbnails directly in the 3D environment. In this figure, the lines connecting the red Trailfinder Weather Tower to webcam thumbnails represent the line-of-sight of the robotic pan/tilt/zoom camera that took the pictures. Those without connecting lines are IR/Visible spectrum cameras inside the nestboxes, monitoring the nesting or roosting birds. By clicking on a thumbnail, as with Nestbox 31 here, you can see a full resolution image.

Figure 38. When the webcam thumbs layer is not directly enabled for viewing in the 3D environment, the user can still click on the polygon representing the site (in this case Nestbox 27), and see the associated cameras available at this location. By clicking on the 'Inside Nestbox' image archives (for the hour, day, week, month of all-time), a web-browser window pane is opened inside Google Earth which display  images from that chosen peroid.  Note the bird roosting overnight at this location.

Figure 39. Sequential Kriging models for the Cold Air Drainage sensor transect. Perhaps one of the most promising advancements of a 3D visualization environment is the use of on-the-fly spatial interpolation of point sensor data (discussed in 'future directions'). Here the Cold Air Transect point data was spatially modeled (using the Kriging technique) every 15 minutes over a 24-hour period. Note the advancement and growth of the colder yellow pockets as the night proceeds. Google Earth currently only support static image overlays, so at the moment the 'animation' of this sequence has to be jury-rigged. However, as Google adds native support for animation, such this secession of images, it can easily illustrate the changes of temperature over time, and the nightly 'cold air drainage' events found in mountain air-sheds.

Figure 40.  Aspect ratio GIS layer (compass direction of the facing slope). GIS layers, shapefiles and polygons outputted from programs such as ESRI's ArcGIS can be displayed in Google Earth. Topographical variables such as slope steepness and aspect can affect micro climate conditions. Consider the red areas above. These areas face north and thus receive less sunlight during the day. The use of basic GIS layers, and more complex ones (such as vegetation and soil-type maps) will be of great use to researchers; this interface will allow users to look at a variety of information types simultaneously. Also note the topographic contour lines in black, drawn as vectors in Google Earth.

Figure 41. Kriging model overlays and graph of fluormetry (measure of aquatic floral density) created from NAMOS buoy and RoboDuck transect data. As data streams from new sensor systems are directly entered into the DMS  database, spatial modeling and access to measurement graphs will be dynamically generated for the Google Earth interface.

Figure 42.  High resolution aerial photos overlayed on top of the James Reserve. Google Earth's default photo overlay quality in the upper right corner. In addition to GIS layers, remote sensing layers can be placed over the James Reserve. This is a 4-inch resolution aerial photograph shown at near-full quality. Such details allows users to identify individual trees and vegetation types. Remote sensing layers from satellites can also be used.

At the moment, the systems and datasets integrated into the Google Earth interface at the James Reserve include the original CMS mote system, weather station, Hobo data loggers, bird biology and plant phenology webcams and a stereo audio feed. Currently the AMARSS, NIMS and NAMOS systems, though represented geographically, do not have their live data streams available in in the Google Earth layer.

GIS analyst Vanessa Rivera Del Rio has made a number of advancements for the James Reserve's Geographic Information Systems library. The high resolution aerial photos, GIS aspect layer and contour maps for Hall Canyon figured above were georeferenced by Vanessa. She has also created slope, elevation, watershed and sun-shade GIS models for the James Reserve. As GIS vegetation maps, and other descriptive geographic information is assembled, the continued interfacing of GIS outputs with Google Earth will be critical.

Accomplishments

A database, SensorBase, has been constructed to allow a common back-end for user interface and data handling.

Non-technical interfaces for NIMS as well as the sensor arrays at JR have been constructed to allow non-technical users access to data and control features.

Future Directions

Incorporation of projects involving Google Earth will expand the local interface work such that users will gain greater access to tools and data.  Work on diagnostic interfaces for mechanical (NIMS) as well as communication parameters (CMS, ESS) will allow users to be involved in debugging and troubleshooting procedures.  Open access to tools are now allowing educational avenues to be explored including student-involved value added content to databases.