Remote Human/Machine Control

a white paper prepared for NASA by

EECS Dept.

Case Western Reserve University

Cleveland, OH 44106

Concept. Intelligent systems, such as robots, are rapidly becoming more competent. However, fully autonomous capability is not on the foreseeable horizon. The simultaneous explosive growth in communications technology offers the opportunity to gain immediate benefits from semi-autonomous systems through shared human/machine control. The communication protocol should support real-time feedback, should be securely integrated with IP-based terrestrial networks, and should allow new components to be seamlessly added. Achieving effective network-based human/machine control would yield benefits in space exploration, in manufacturing, and in domestic automation.

Motivation. Dramatic, continuing advances in communications are anticipated, both on Earth and in space. Expanding bandwidth and connectivity provides a foundation for opportunity, but reaping the rewards will depend on how we engineer and manage the flow of information. One of the opportunities to exploit advanced communications, both on Earth and in space, is in the context of human/machine interfacing with intelligent systems. Such communications could enable dramatic advances in space exploration, factory automation, and domestic robotics. One of the most recognizable forms of intelligent systems is robotics, though the issues in intelligent robots are common to other types of intelligent systems. Use of robots is growing both on Earth and in space, in large part due to increased capacity for machine intelligence. In industry, as in space exploration, acceptance of robotics requires very high reliability. A fundamental barrier to high reliability is the occurrence of unexpected events and the ability of the intelligent system to handle errors or anomalies autonomously and appropriately. Enhanced communications offers a means to address this challenge.

Creating a foolproof system that can operate successfully in unstructured and (at least partially) unpredictable environments is an unrealistic goal for the next century. However, much progress can be made using systems with more limited competence, assisted by human interactions. An extreme example of such interactions is the shuttle arm. Such human/machine interaction involves full concentration for direct real-time control. In this case, the machine is teleoperated, having no local intelligence of its own. Direct, full-time human intervention is useful and practical when communication time delays are sufficiently short, when the task addressed is highly valued, and when direct manual operation is precluded (e.g., hazardous environments, hazardous material handling, remote operations). However, due to communication time delays, direct teleoperation is impractical even from Earth to Earth orbit. Opportunities in industry for application of teleoperation are also limited.

Between the extremes of real-time teleoperation vs. fully autonomous intelligent systems there is a spectrum of control options involving humans as supervisors of the machines. An example is the Mars Rover, which was able to operate with local intelligence while receiving higher-level instructions from Earth. In this manner, a human interacting with the machine augments its competence by relying on the versatile error-handling capacity of the human and the limited local intelligence of the machine. Rather than wait for the ultimate results of research in competent, fully autonomous intelligent systems, we can realize benefits from a continuum of increasingly intelligent machines, assisted by humans. Through this effort in the High-Rate Data Delivery communications thrust, advanced communications would leverage and augment the efforts of NASA's Thinking Systems Thrust Area (http://cetdp.jpl.nasa.gov/ta-9.html). A machine with limited intelligence would be able to perform some tasks autonomously, but would be impeded by numerous unanticipated situations. In our proposed scenario, such a machine would contact a human for help via Internet communications. The remote human would assess the situation and respond with appropriate error handling. In this manner, capital-intensive investments could be made practical by keeping them productive, presumably with relatively low demand on the time of remote human experts.

Industrial applications. Examples of the current demand for Internet-based human/machine interaction are illustrated by local anecdotes. Foseco, Inc., a Cleveland-based company, provides a service of dispensing refractory coatings for Bethlehem Steel in Burns Harbor, Indiana. Their installed system uses robots for applying the coatings, which is yielding higher quality and lower waste. However, when the system encounters a technical problem, a Foseco representative must travel from Cleveland to Burns Harbor to resolve the problem. This burden limits the attractiveness of Foseco's system and limits their sales opportunities geographically. A sufficiently sophisticated Internet-based interface would enable Foseco to monitor, reset, or modify systems remotely, improving productivity, profitability, and market opportunities.

Another local example comes from Lincoln Electric - a Cleveland-based company with an international reputation for excellence in welding. They currently have a strong market share in robotic welding, and they protect their position in the market with superior service. It is their hope to achieve higher levels of service through Internet-based communications. Lincoln would like to monitor the operation and health of their fielded systems in order to recommend preventive maintenance, to anticipate and recommend materials restocking, and to advise on possible applications programming improvements. Internet-based communications with their fielded systems would enable their service to be proactive rather than reactive.

Text Box: Space-based applications. A number of robots will be deployed in space to perform, for example, experiments or spacecraft maintenance on the International space station, on Mars, or in deep space. Areas of applications include spacecraft maintenance and construction of bases on Mars and in space. Space construction or maintenance is an especially sensitive area that involves risks for the physical safety of humans, and it can be appropriately delegated to robots. Robots can perform experiments on behalf of an Earth-based investigator who controls the robots through the combination of long-haul space links, terrestrial gateways, and a terrestrial network. The combination of a communication protocol with space-based actuators and sensors results in the virtual presence of the investigator team in space.

Long-term applications. These examples are illustrative of immediate needs that could be addressed with current or near-term technology. More futuristically, we can imagine more sophisticated intelligent machines presented with more challenging situations. Domestic robots are an example. A fully competent autonomous domestic robot is far from technically practical in the foreseeable future. However, robots with limited competence could be practical if they could be kept productive. A possible means to do so is by recruiting remote assistance from owners via the Internet. At CWRU, we have demonstrated an exploratory case of a robot that performs an example household task (sorting laundry). When the robot is confused by some situation or outcome, it contacts a remote human via the network. Via browser interactions (see figure), the remote human is able to assess the situation via pictures, can instruct the robot to perform diagnostic or functional actions, and can make decisions that enable the robot to return to productive work. The project is drawing from an inter-disciplinary background in automation (Prof. Newman) and client-server architectures (Prof. Liberatore).

Communication protocol. It is our hope that enhanced communications will enable intelligent systems to be economically practical immediately. The key to achieving the objective is to establish a foundation for Internet-based human/machine interaction. In the first place, a communication protocol is needed for the control center to communicate with a unit and resolve the unit’s control conflicts. The protocol will contain transport and application layer solutions to address the needs of remote control. The major objectives for such protocol are:

Coarse-grained control. On-board control is neither fully autonomous nor completely dependent on human input. As a result, the protocol must support a coarse-grained style of interaction between the remote human user and the robot.

Flexible interaction. The protocol should support a variety of interaction methods ranging from remote commands to question/answer dialog and reprogramming from a distance.

Real-time OS. The on-board control can be supported by a real-time operating system and software, with which the communication protocol must be integrated.

Real-time media. The feedback from sensors in space will be mostly images, videos, and other real-time media. The feedback must be rendered faithfully to the human user so that she can interpret correctly the system behavior and take the appropriate corrective steps. By contrast, inaccurate or jittery feedback would be disorienting to a human engineer and make it difficult for her to diagnose correctly the robot’s problem. The communication protocol must support realistic feedback.

Tether-free environments. Mobile robots might need to roam around their operational environments. The protocol needs to support these applications and provide for wireless communication channels to the robot.

Interoperability. An investigator located on Earth can be linked to the space robots and sensors through commercial gateways and terrestrial networks. Since terrestrial networks are mostly IP-based, it is critical that the control protocol be IP-based or IP-compatible. Analogously, an investigator might want to try control commands on a mock-up on Earth before issuing them to a space-based robot. If the mock-up is not locally accessible to the investigator, it can be tried only through a terrestrial network, presumably IP-based.

Security. If communication crosses an untrusted IP network, the protocol must guarantee that no malicious third party can access commands or feedback information.

Extensibility. The protocol should support the fast and seamless addition of new components, such as new sensors and actuators (plug-and-play).

Approach. Our approach is to leverage on existing industrial protocols and to adapt them for the purposes of NASA enterprises. In the first phase of the project, we intend to deploy existing protocols, to examine to what extent their design choices are appropriate for the remote control of robots in space, and to establish which new challenges need to be addressed. An additional area of investigation is the concern for safety.

As an example investigation, we propose to perform emulation of robotically controlled experiments on the international space station. With application-domain recommendations from NASA and utilizing robotic and networking facilities at CWRU, we will construct a mock-up of a robotically-tended space-based laboratory. We will develop, test and evaluate interfaces for shared human/machine control of remote experiments to be performed by this system.

As part of this effort, we will seek a collaboration with a local company (likely, one of those listed above). In this collaboration, we seek to import industrial expertise and market relevance, and to give back to industry in terms of new network-based capabilities.