Overview

The Disruption Tolerant Shell (DTS) is a distributed system management tool in the final stages of development. The target systems for the tool are large wireless sensor networks of PC and PDA class devices which can be characterized by poor link quality, frequent connectivity disruptions, or duty cycled nodes.

The initial development of DTS was driven by the the MesoAmerican Subduction Experiment (MASE) seismic broadband array. The MASE seismic broadband array consists of 100 seismic stations covering 500 km. of which 50 are equipped with 802.11b wireless network interface cards and directional antennae. The sites are up to 20 km apart which causes the majority of the network topology to be identical to the physical topology: each station can only communicate with the next up stream and the next down stream station. The exceptions are a few star topologies where a few stations share a similar down stream neighbor and a few small clusters of interconnected stations.

Management of the network initially proved difficult because of the poor link quality, frequent disruptions, and inability to create end to end connections beyond a few hops. Occasionally it is possible to use standard Unix remote access tools to connect hop by hop to each of the nodes in the network, but the majority of the time it is impossible to reach all the nodes. This causes the latency of system management to be very large as is the amount of attention required by the user to complete any management task.

The connectivity among the stations is generally poor. The mountainous terrain combined with strict requirements on where the sensor stations are located contributes to the poor connectivity. The connectivity is enough to get the data through but the poor quality of the individual links and frequent disconnections prevent data collection to happen in an end to end fashion. Typically it is not possible to create and maintain end to end connections using TCP for more than 3 or 4 hops.

Approach

Data deliver on the MASE seismic array is addressed by using Disruption Tolerant Networking (DTN) techniques. Each node generates 1-3MB of data per hour. The data is stored in hour long files called bundles. The bundles are deliberately transferred hop by hop towards the sink. The path to the sink is determined by using the ETX metric and forming a sink tree. Each node maintains custody of the locally generated bundle until an acknowledgment from the data sink is received.

The management issue is addressed by DTS. DTS provides the user an asynchronous shell interface to every node in the network simultaneously. The user can input blocks of commands to be executed on every node. The outputs from the commands are disseminated to all the other nodes in the network. Since there is no guarantee of in order arrival of the commands DTS will only execute commands in the order they are issued at the shell.

The commands and the output are treated as shell state and each node collects and remembers as much of the state as it can. In other words, all the commands that have been issued and all the output from the commands stay on all the nodes, even after execution. This provides a basic form of robustness if a neighboring node is down or a link is temporarily disrupted: when a link comes back up it will receive the commands from one of its neighbors, possibly execute them, and then pass the commands along to nodes further up the line. The same applies to the output from the commands: the output eventually disseminates to all the nodes. Compared to an end to end method of executing commands on a node, there is no direct interaction between the node issuing the command and all the nodes in the network. This decreases the latency for a single command to be executed on all the nodes because the node issuing the commands does not have to search for the moment when an end to end connection might exist to a random node in the network for all the nodes in the network.

The persistence of shell state and the shell interface itself allows the user to combine their knowledge about the basic topology of the network to track the progress of any command through the network. In addition, they can implicitly determine the status of various links, detecting situations such as partitions or faulty nodes. Further robustness is provided by each node saving the commands and responses to disk. The commands and output remains in the network until the user chooses to remove them from the shell state.

DTS also has a few specialized commands. One allows the user to push files to all the nodes in the system. This command is treated just like a standard shell command, so it can be used to push a file to all the nodes and then execute a standard command that depends on that file. The other specialized command is a status command which creates a permanent channel for status information from the Emstar software framework that DTS is built with. The output of the status command is updated periodically so changes in the status are disseminated to the network. They can provide log information, state, or statistics from the specific components that make up the DTS application.

DTS is implemented on top of a reliable state dissemination mechanism called statesync. Statesync provides a publish-subscribe interface where a component can publish a table with keyed entries. Changes to published tables are implemented as logs on the table entries. The logs are checkpointed to help keep overhead down. Each node maintains a sequence number representing its latest state. If a neighboring node has a different sequence number for the node, it can ask for retransmissions of the specific logs to catch up with the state. This provides increased efficiency over simply retransmitting the entire table to synchronize the nodes. The reliability is achieved through the retransmission protocol.

Systems/Experiments

DTS has been simulated using the Emstar framework. It has also been tested on multiple testbeds in the CENS System Lab. The DTS system will be installed on the MASE seismic array. We will run experiments comparing the latency of DTS and end to end management solutions on local testbeds and the MASE seismic array.

Accomplishments

• Characterization of connectivity and data collection using prior methods on the MASE seismic array
• Design and implementation of a DTN transfer mechanism
• Design and implementation of DTS

Future Directions

We plan on running extensive experiments on real networks such as the MASE seismic array, future PDA and PC class deployments, and future duty cycled testbeds.

Networks with duty cycling nodes are very similar to networks that have poor connectivity and frequent disruptions. We are developing a testbed of duty cycling nodes on which we can evaluate the feasibility of DTS for this class of networks.

DTS currently provides the ability to issue almost any shell command that standard Unix shells and remote access tools accept. This creates the potential for disruption the entire operation of the network with an invalid or unfriendly command. We plan on developing better shell semantics that remove the potential to break the system. Some very simple examples of this which have already been implemented are: no shell expansion through the use of operators such as '*' and restricting all path names to be absolute.

People

• Graduate Students: Martin Lukac; Lewis Girod; Andrew Parker; Allen Husker
• Staff: Igor Stubailo; Richard Guy
• Faculty: Paul Davis, Geophysics; Deborah Estrin, Computer Science