Apparatuses, Systems, and Methods for Apparatus Operation and Remote Sensing

• US 20110066262A1
• Filed: 01/22/2009
• Published: 03/17/2011
• Est. Priority Date: 01/22/2008
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Rander, Peter William, Warner, Randon, Kelly, Alonzo James, Herman, Herman
• Granted Patent: US 8,774,950 B2
• Time in Patent Office: Days
• US Class Current: 700/90
• CPC Class Codes: G05B 23/0267 Fault communication, e.g. human machine interface [HMI]

Abstract

A method and system for controlling an apparatus including receiving data indicative of an actual state of the apparatus, defining a first viewpoint relative to at least one of the environment and the apparatus, determining a first predicted state of the apparatus at time T, determining a first predicted state of the environment at time T, producing a first virtualized view from the first viewpoint, sending a first control signal to the apparatus after producing the first virtualized view, defining a second viewpoint relative to at least one of the apparatus and the environment, determining a second predicted state of the apparatus at time T+delta T, determining a second predicted state of the environment at time T+delta T, producing the second virtualized view from the second viewpoint, sending a second control signal to the apparatus after producing the second virtualized view, and changing the actual state of the apparatus based on the first control signal.

Claims

1. A method for controlling an apparatus (12) in an environment (18), comprising:

receiving (810) data indicative of an actual state of the apparatus (12);

defining (815) a first viewpoint relative to at least one of the environment (18) and the apparatus (12);

determining (820) a first predicted state of the apparatus (12) at time T, wherein T is current time plus additional time representative of latency for a control signal to be received and be implemented by the apparatus (12), and wherein the first predicted state of the apparatus (12) is determined from at least one previous actual state of the apparatus (12);

determining (825) a first predicted state of the environment (18) at time T;

producing (830) a first virtualized view from the first viewpoint, wherein the first virtualized view uses encoded data, wherein the first virtualized view is indicative of both the first predicted state of the apparatus (12) at time T and the first predicted state of the environment (18) at time T;

sending (835) a first control signal to the apparatus (12) after producing the first virtualized view;

defining (840) a second viewpoint relative to at least one of the apparatus (12) and the environment (18), wherein defining a second viewpoint occurs after defining a first viewpoint;

determining (845) a second predicted state of the apparatus (12) at time T+delta T, where delta T is a difference in a time between displaying the first virtualized view and a second virtualized view, and wherein the second predicted state of the apparatus (12) is estimated from at least one previous actual state of the apparatus (12) and from at least one previous control signal to the apparatus (12);

determining (850) a second predicted state of the environment (18) at time T+delta T;

producing (855) the second virtualized view from the second viewpoint, wherein the second virtualized view uses encoded data, and wherein the second virtualized view is indicative of both the second predicted state of the apparatus (12) at time T|delta T and the second predicted state of the environment (18) at time T+delta T;

sending (860) a second control signal to the apparatus after producing the second virtualized view; and

changing (865) the actual state of the apparatus (12) based on the first control signal.

2. The method of claim 1, wherein the second predicted state of the apparatus (12) is estimated from at least one predicted state of the environment (18).

3. The method of claim 2, wherein the second predicted state of the apparatus (12) is estimated from geometry of at least one predicted state of the environment (18).

4. The method of claim 1, further comprising producing the encoded data, wherein producing encoded data includes:

receiving (910) data indicative of a plurality of different representations of a portion of the environment (18);

identifying (920) corresponding elements in the different representations of the portion of the environment (18); and

creating (930) encoded data representative of the portion of the environment (18).

5. The method of claim 4, wherein receiving (910) data indicative of a plurality of different representations of a portion of the environment (18) includes receiving (910) data from a sensor (24) on the apparatus (12).

6. The method of claim 4, further comprising:

storing (940) the encoded data in a database after creating (930) the encoded data; and

wherein;

producing (830) a first virtualized view includes retrieving the encoded data from the database; and

producing (855) the second virtualized view includes retrieving the encoded data from the database.

7. The method of claim 4, wherein creating (930) encoded data includes encoding appearance and geometry of the portion of the environment (18).

8. The method of claim 7, wherein creating (930) encoded data includes encoding the appearance of a portion of the environment (18) to be non-photorealistic.

9. The method of claim 7, wherein creating (930) encoded data includes encoding the appearance of a portion of the environment (18) to display the portion of the environment (18) in a false color to represent additional information about the portion of the environment.

10. The method of claim 9, wherein creating (930) encoded data includes encoding the appearance of a portion of the environment (18) to represent a portion of the environment (18) that is not visible to an unaided human eye.

11. The method of claim 4, wherein the encoded data is produced with data from sensors (24) on the apparatus (12).

12. The method of claim 4, wherein the encoded data is produced with data from a sensor that is not on the apparatus (12).

13. The method of claim 1, wherein the estimated latency for a control signal to get to the apparatus (12) is estimated latency between the control signal being initiated at the control agent (14) and the control signal being implemented by the apparatus (12).

14. The method of claim 1, wherein the estimated latency for a control signal to get to the apparatus (12) is estimated latency between a user providing an input at the control agent (14) that results in a control signal being sent to the apparatus, and the apparatus (12) implementing the control signal.

15. The method of claim 1, wherein the first and second control signals include encoded data.

16. The method of claim 1, wherein the first and second control signals include control signals specifying movement and direction commands to be implemented by the apparatus (12).

17. The method of claim 1, wherein the first and second control signals include control signals specifying a position and orientation to be achieved by the apparatus (12) at a time in the future.

18. A system (10) for operation in an environment (18), comprising:

an apparatus (12) including a sensor (24);

a control agent (14);

a processor (20, 40) connected to at least one of the apparatus (12) and control agent (14);

a memory device (22, 42) connected to the processor (20, 40), wherein the memory (22, 42) includes computer-readable instructions which, when executed by the processor (20, 40), cause the processor (20, 40) to perform the steps of;

receiving (810) data indicative of an actual state of the apparatus (12);

defining (815) a first viewpoint relative to at least one of the environment (18) and the apparatus (12);

determining (820) a first predicted state of the apparatus (12) at time T, wherein T is current time plus additional time representative of latency for a control signal to be received and implemented by the apparatus (12), and wherein the first predicted state of the apparatus (12) is estimated from at least one previous actual state of the apparatus (12);

determining (825) a first predicted state of the environment (18) at time T;

producing (830) a first virtualized view from the first viewpoint, wherein the first virtualized view uses encoded data, wherein the first virtualized view is indicative of the first predicted state of the apparatus (12) and the environment (18) at time T;

sending (835) a first control signal to the apparatus (12) after producing the first virtualized view;

defining (840) a second viewpoint relative to at least one of the apparatus (12) and the environment (18), wherein defining a second viewpoint occurs after defining a first viewpoint;

determining (845) a second predicted state of the apparatus (12) at time T+delta T, where delta T is a difference in a time between displaying the first virtualized view and a second virtualized view, and wherein the second predicted state of the apparatus (12) is estimated from previous actual states of the apparatus (12) and from previous control signals to the apparatus (12);

determining (850) a second predicted state of the environment (18) at time T;

producing (855) the second virtualized view from the second viewpoint, wherein the second virtualized view uses encoded data, and wherein the second virtualized view is indicative of the second predicted state of the apparatus (12) and the environment (18) at time T+delta T;

sending (860) a second control signal to the apparatus after producing the second virtualized view; and

changing (865) the actual state of the apparatus (12) based on the first control signal.

19. The system (10) of claim 18, wherein the control agent (14) is part of the apparatus (12).

20. The system (10) of claim 19, wherein the control agent (14) and the apparatus (12) share a common processor (20, 40) and share common memory (22, 42).

21. The system (10) of claim 18, wherein:

the apparatus (12) includes a processor (20), memory (22), a sensor (24), and a transmitter/receiver (28);

the control agent (14) includes a processor (40), memory (42), and a transmitter/receiver (48); and

the apparatus (12) and the control agent (14) are separate from each other and connected by a communications link (16) between the transmitter/receiver (28) in the apparatus (18) and the transmitter/receiver (48) in the control agent (14).

22. The system (10) of claim 21, further comprising:

a vehicle controller (26) connected to the processor (20) in the apparatus (12);

an input device (44) connected to the processor (40) in the control agent (14); and

an output device (46) connected to the processor (40) in the control agent (14).