Tele-autonomous system and method employing time/position synchrony/desynchrony

• US 5,046,022 A
• Filed: 03/10/1988
• Issued: 09/03/1991
• Est. Priority Date: 03/10/1988
• Status: Expired due to Term

• Current Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Original Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Inventors: Conway, Lynn A., Volz, Richard A., Walker, Michael W.
• Primary Examiner(s): MacDonald, Allen
• Application Number:
  US07/166,337
• Time in Patent Office: 1,272 Days
• Field of Search: 364/513, 364/190, 364/167.01, 364/424.02, 901/9, 901/50, 901/10, 180/167-169, 318/580, 318/587, 318/568.2, 318/568.23
• US Class Current: 700/250
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 19/427 Teaching successive positions by tracking the position of a joystick or handle to control the positioning servo of the tool head, master-slave control G05B19/423 takes precedence

Abstract

Systems and methods for performing tele-operations with the active assistance and supervision of a cognitive agent, such as a human being, who generates a future path plan in real time for immediate use by an automated on-line controlled agent, such as a work robot or other manipulator, are disclosed. The systems and methods may employ a visual display to present images of the controlled agent, of a foward simulation of the controlled agent, and of the future path plan being generated to enable a human operator to assess and control the on-going activity. The system simultaneously operates the controlled agent and produces the forward simulation and path plan, all under real time conditions. The cognitive agent, display and forward simulation equipment may be at one location and the controlled agent and its controller at another remote location. The forward simulation may be selectively operated in one of three modes: a time-and-position synchronized mode, a position-synchronized, time-desynchronized mode, and a mode where neither time nor position are synchronized with the controlled agent. The desynchronization of time and of position of the forward simulation with respect to the controlled agent is accomplished by use of a time clutch and a position clutch. The forward simulation can also be reversed through the use of a time brake, and the time ratio between the forward simulation time and real time can be adjusted. Finally, a method for transferring active control of functioning autonomously-operated equipment from one cognitive agent to another cognitive agent under real time conditions is disclosed.

Claims

1. In a system for enabling a cognitive agent to operate in real time a controlled agent to perform automatically movements corresponding to a sequence of command data stored in the system representing a planned path in space and time of at least a selected portion of the controlled agent, a method of generating the sequence of command data used to control the controlled agent, comprising the steps of:

(a) providing first input means operable by the cognitive agent for producing in real time successive command signals that specify the planned path in space and time of at least the selected portion of the controlled agent;

(b) providing second input means operable by the cognitive agent for selectively enabling and disabling time synchrony;

(c) generating command data to be added to the sequence of command data representing the planned path by deriving from the successive command signals at predetermined intervals of time information related to desired position, and sequentially storing such derived positional information as command data in a manner indicating relative order of acquisition, thereby providing time synchronized path planning;

(d) when the time synchrony is enabled, also storing the derived positional information in a manner which indicates relative time of acquisition, as well as relative order of acquisition; and

(e) when the time synchrony is disabled, deriving from the command signals at predetermined intervals of time information related to desired position by also storing the derived positional information in a manner which indicates its independence from its relative time of acquisition.

2. A method of generating command data, as in claim 1, further comprising the step of:

(f) automatically generating, in a manner substantially independent of relative time of acquisition, time values for the sets of command data derived during an interval of time when time synchrony is disabled, thereby providing an artificial time base for portions of sequence of command data specifying the planned path generated when time synchrony is disabled.

3. A method of generating command data, as in claim 1, wherein:

the cognitive agent is a human being, and the controlled agent is a work robot or other manipulator having a plurality of power-driven signal-controlled links interconnected to permit relative motion therebetween to defined plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system, and the method further comprises the steps of;

(f) providing a monitor for visually displaying information;

(g) displaying on the monitor the relative location of at least the selected portion of the manipulator with respect to a scene thereon; and

(h) displaying on the monitor simultaneously with step (g) that portion of the planned path generated but not yet performed by the manipulator.

4. A system for concurrently driving, in real time by a cognitive agent, a controlled agent movable in at least two dimensions to perform automatically movements corresponding to a sequence of command data stored in the system representing a planned path of at least a selected portion of the controlled agent, said system comprising:

first input means operable by the cognitive agent for producing in real time successive command signals that specify the planned path of at least the selected portion of the controlled agent;

means for generating command data to be added to the sequence of command data representing the planned path by deriving from successive command signal information related to desired position of at least the selected portion of the controlled agent;

memory means for successively storing derived command data in sets in a manner which indicates order of receipt thereof;

second input means operable by the cognitive agent for selectively enabling and disabling the time synchrony; and

control means for enabling and disabling time synchrony of path planning by selectively altering, in response to the second input means, the timing of the derived command data relative to the time of acquisition of the command signals from the first input means.

5. A system as in claim 4, further comprising:

forward simulation means for providing a visual presentation of at least a portion of the planned path not yet executed by the controlled agent.

6. A system as in claim 5, wherein:

the cognitive agent is a human being;

the controlled agent is a work robot or other manipulator having a plurality of power-driven, signal-controlled links interconnected to permit relative motion therebetween to define plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system,

the forward simulation means includes a visual display mounted for viewing by the human being, and

the memory means for successively storing derived command data in sets includes a queue of memory locations which may be operated in a first-in, first-out fashion.

7. A system as in claim 6, further comprising:

remote control means for enabling the manipulator to be located and operated remotely from the location of the human being and the forward simulation means, the remote control means including a two-way communications link between the location of the manipulator and the location of the human being.

8. In a system for enabling cognitive agent to operate in real time a controlled agent to perform automatically movements corresponding to a first sequence of command data stored in the system representing a planned path in space and time of at least a selected portion of the controlled agent, a method of processing command signals to generate the first sequence of command data used to control the controlled agent, comprising the steps of:

(a) providing first input means operable by the cognitive agent for producing in real time a stream of command signals that specify the planned path in space and time of at least the selected portion of the controlled agent;

(b) providing second input means operable by the cognitive agent for selectively enabling and disabling the continuity of position in path planning;

(c) when continuity of position is enabled, generating command data to be added to the first sequence of command data representing the planned path by deriving from successive command signals at predetermined intervals of time information related to desired position, and sequentially storing such position information as command data associated with the planned path, thereby providing continuity of position in path planning;

(d) when the continuity of position is disabled, interrupting the generation of the position command data representing the planned path; and

(e) when the continuity of position is disabled, generating from the stream of command signals a second sequence of position command data specifying a possible future path plan of at least the selected portion of the controlled agent.

9. A method as in claim 8, further comprising the steps of:

(f) depicting, on a visual display a scene representing at least selected attributes of an environment in which the controlled agent is operating;

(g) depicting on the visual display the current position of at least a selected segment of the controlled agent in relation to the selected attributes;

(h) generating a first forward simulation representing the planned path in space and time indicated by the command data generated during step (b) of at least a selected segment of the controlled agent in relation to the existing position of the selected segment shown in the scene;

(i) depicting, on the visual display as part of the scene, a second forward simulation representing the possible future path plan generated in connection with step (e).

10. A method as in claim 9, wherein:

the cognitive agent is a human being;

the controlled agent is a work robot or other manipulator having a plurality of power-driven links interconnected to permit relative motion therebetween to define plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system.

11. A system for concurrently driving, under closed-loop servo control alterable in real time by a cognitive agent, an articulated work robot or other manipulator having a plurality of power-driven links interconnected to permit relative motion therebetween in plural degrees of freedom, the links being controlled automatically to perform movements corresponding to a sequence of command data stored in the system representing a planned path of at least a selected portion of the manipulator, said system comprising:

first input means operable by the cognitive agent for producing in real time successive command signals that specify the planned path of at least a selected portion of the manipulator;

means for generating command data to be added to the sequence of command data representing the planned path by deriving from successive command signals, information related to desired positions of at least selected portion of the manipulator;

second input means operable by the cognitive agent for selectively enabling and disabling the position synchrony of path planning;

first memory means for successively storing derived command data in a manner which indicates intended order of processing under closed-loop servo control;

control means for interrupting the storing of derived command data in the first memory means;

second memory means for storing command data during the interval of interruption and in a manner indicating the order of receipt thereof; and

forward simulation means for providing a visual presentation of the desired postions represented by the derived command data stored in the second memory.

12. A system as in claim 11, further comprising:

means for automatically generating positional command data to link the most recently received command data in the second memory means with the most recently received command data in the first memory means upon re-enablment of position synchrony.

13. A system as in claim 12, further comprising:

first buffer means associated with the forward simulation means for holding a stream of derived command data;

remote control means controlling the automatic operation of the manipulator, said remote control means being located remotely from the forward simulation means;

second buffer means associated with the remote control means for holding a stream of command data for delivery to the remote control means;

data communication means for providing a first communication link between the first and second buffer means and a second communication link between the forward simulation means and remote control means; and

means for moving data from the second memory means to the first buffer means upon re-enablement of position synchrony.

14. In a system for interactively allowing a cognitive agent to operate in real time, a controlled agent to perform automatically movements corresponding to a first sequence of command data stored in the controller representing a planned path in space and time of at least a selected portion of the controlled agent, a method of processing command signals to generate the first sequence of command data used to control the controlled agent, comprising the steps of:

(a) producing a stream of command signals in real time that specify the planned path in space and time of the selected portion of the controlled agent; and

(b) selectively reversing the path planning by successively deleting only a portion of the command data from the sequence of command data not yet performed by the controlled agent, the command data in the sequence being deleted on a last-in, first-out basis.

15. A method as in claim 14, further comprising the step of:

(c) specifying a rate at which data is to be deleted within a range of allowed rates of deleting command data from the sequence.

16. A method as in claim 14, further comprising the step of:

(c) providing third means operable by the cognitive agent for deleting in a substantially simultaneous manner, all command data in the sequence not yet performed by the controlled agent.

17. A method as in claim 14, wherein:

the cognitive agent is a human being,

the controlled agent is a work robot or other manipulator having a plurality of power-driven, signal-controlled links interconnected to permit relative motion therebetween to define plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system, and the method further comprises the step of;

(c) generating, while on-line with the system by use of the first input means, a future planned path in space and time of the selected portion of the controlled agent;

(d) depicting, on a visual display a scene representing at least selected attibutes of an environment in which the controlled agent is operating;

(e) depicting on the visual display the current position of at least a selected segment of the controlled agent in relation to the selected attributes;

(f) generating a first forward simulation representing the future planned path in space and time generating during step (c);

(g) selectively reversing, while on-line with the system by use of the second input means, the future path planning by selectively deleting command data from the sequence of command data; and

(h) altering the first forward simulation generated during step (f) to reflect the deletions of command data during step (g).

18. A system for enabling a cognitive agent to operate in real time a controlled agent to perform automatically movements corresponding to a first sequence of command data stored in the system representing a planned path in space and time of at least a selected portion of the controlled agent, the system comprising:

first means operable by the cognitive agent for producing a stream of command signals in real time that specify a future planned path in space and time of the selected portion of the controlled agents; and

second means operable by the cognitive agent for selectively reversing the path already planned by successively deleting command data from the sequence of command data representing that portion of the future path plan not yet performed by the controlled agent, the command data in the sequence being deleted on a last-in, first-out basis.

19. A system as in claim 18, wherein:

the cognitive agent is a human being,

the controlled agent is a work robot or other manipulator having a plurality of power-drive, signal-controlled links interconnected to permit relative motion therebetween to define plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system, and system further comprises;

first memory means for successively storing derived command data in sets includes a queue of memory locations operable in a first-in, first-out fashion;

means for specifying a rate at which data is to be deleted within a range of allowed rates of deleting command data from the sequence; and

forward simulation means for providing a visual presentation of at least a portion of the future planned path not yet executed by the controlled agent.

20. A system as in claim 18, further comprising:

remote control means for enabling the controlled agent to be located and operated remotely from the location of the cognitive agent and the forward simulation means, said remote control means including telecommunications channels for providing two-way transfer of information between the location of the controlled agent and the location of the cognitive agent.

21. In a system for interactively allowing a cognitive agent to operate in real time a controlled agent to perform automatically movements corresponding to a sequence of command data stored in the system representing a planned path in space and time of at least a selected portion of the controlled agent, a method of generating the sequence of command data used to control the controlled agent, comprising the steps of:

(a) providing first means operable by the cognitive agent for producing a stream of command signals in real time that specify the planned path in space and time of the selected portion of the controlled agent; and

(b) selectively generating command data to be added to the sequence of command data representing the planned path by acquiring from the stream of command signals at predetermined intervals of time position command information and successively storing with such position command data indicating that such stored command data is to be performed in time by the controlled agent at a non-unity scaled function of time of acquisition, thereby providing a planned path having positional information synchronized to a time scale different from but proportional to the actual time of acquisition of such positional information.

22. A method as in claim 21, further comprising the steps of:

(c) selectively generating command data to be added to the sequence of command data representing the planned path by acquiring from the stream of command signals at predetermined intervals of time sets of position command information and sequentially storing such position command information in a manner indicating time of acquisition, thereby providing time synchronized path planning;

(d) providing second means operable by the cognitive agent for selecting whether the path planning shall be executed by the controller in accordance with step (b) or step (c).

23. A method as in claim 21, further comprising the steps of:

(c) providing a second input means operable by the cognitive agent for specifying a particular time scale selectable from an allowed range of different time scales.

24. A method as in claim 23, wherein:

the cognitive agent is a human being, and the controlled agent is a work robot or other manipulator having a plurality of power-drive, signal-controlled links interconnected to permit relative motion therebetween to define plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system, and the method further comprises the steps of;

(d) providing a second means operable by the cognitive agent for specifying a particular time scale selectable from an allowed range of different time scales;

(e) providing a monitor for visually displaying information;

(f) displaying on the monitor the relative location of at least the selected portion of the controlled agent with respect to a scene thereon; and

(g) displaying on the monitor simultaneously with step (g) that portion of the planned path generated but not yet performed by the controlled agent.

25. A system for enabling a cognitive agent to operate in real time a controlled agent to perform automatically movements corresponding to a first sequence of command data stored in the system representing a planned path in space and time of at least a selected portion of the controlled agent, the system comprising:

first means operable by the cognitive agent while the controlled agent is automatically performing movements for producing a stream of command signals in real time that specify the future planned path in space and time of the selected portion of the controlled agent;

second means for selectively generating command data to be added to the sequence of command data representing the planned path by acquiring from the stream of command signals at predetermined intervals of time position command information; and

third means for successively storing such position command information and for storing with such position command information, other information indicating that such stored command information is to be performed in time by the controlled agent at a non-unity scaled function of time of acquisition, thereby providing a planned path having positional information synchronized in a time scale different from but proportional to the actual time of acquisition of such position command information.

26. A system as in claim 25, further comprising:

means for selectively generating command data to be added to the sequence of command data representing the planned path by acquiring from the stream of command signals at predetermined intervals of time positional command information and sequentially storing such position command information in a manner indicating time acquisition, thereby providing time-synchronized path planning; and

means operable by the cognitive agent for selecting whether the path planning shall be executed by the system in a time-synchronized manner or a non-unity time-scaled manner.

27. A system a sin claim 26, wherein:

the cognitive agent is a human being, and the controlled agent is a work robot or other manipulator having a plurality of power-driver, signal-controlled links interconnected to permit relative motion therebetween to defined plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system, and the system further comprises;

means operable by the cognitive agent for specifying a particular time scale selectable from an allowed range of different time scales;

monitor means for visually displaying information depicting the relative location of at least the selected portion of the controlled agent with respect to a scene also displayed thereon; and

means for displaying on the monitor means simultaneously with the information depicting the relative location of at least the selected portion of the controlled agent, that portion of the planned path generated but not yet performed by the controlled agent.

28. In a real time control application involving subjecting autonomously-operated equipment to external control by one of a plurality of cognitive agents, a method of providing control transitions wherein external control is passed from a first one of the cognitive agents to a second one of the cognitive agents, the method comprising the steps of:

(a) providing first means operable by the first cognitive agent for producing a stream of first command signals in real time to specify a first portion of the planned path in space and time of at least the selected segment of the equipment;

(b) generating, from the stream of first command signals, a first forward simulation representing the planned path in space and time of at least the selected segment of the equipment in relation to the existing position of the selected segment, said first forward simulation being produced in response to commands received from the first cognitive agent;

(c) providing second means operable by the second cognitive agent for producing a stream of second command signals in real time to specify a second portion of the planned path in space and time of the at least selected portion of the equipment;

(d) generating, from the stream of second command signals, a second forward simulation representing a possible location in the scene for at least the selected segment of the equipment; and

(e) transferring external control from the first agent to the second agent when the second simulation is within a predetermined distance from a portion of the planned path of the first simulation.

29. A method as in claim 28, wherein:

at least one of the cognitive agents is a human being, and the equipment is a work robot or other manipulator having a plurality of power-driven, signal-controlled links interconnected to permit relative motion therebetween to define plural degrees of freedom, the links being manipulated automatically to perform the movements corresponding to the sequence of command data stored in the system.