3-dimensional telepresence system for a robotic environment

• US 6,148,100 A
• Filed: 12/22/1997
• Issued: 11/14/2000
• Est. Priority Date: 12/20/1996
• Status: Expired due to Fees

• Current Assignee: Battelle Energy Alliance LLC (Battelle Memorial Institute)
• Original Assignee: Bechtel BWXT Idaho, LLC
• Inventors: Anderson, Matthew O., McKay, Mark D.
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US08/996,015
• Time in Patent Office: 1,058 Days
• Field of Search: 382/103, 382/107, 382/154, 382/153, 901/1, 901/46, 901/50, 348/211, 348/113-116, 348/20, 348/15, 348/139, 348/143, 348/159, 348/36, 348/47, 348/67, 318/567, 318/568.1, 318/568.11, 318/568.12, 318/568.13, 318/628, 345/419, 345/427, 345/156-158, 345/168, 345/179, 700/245-264, 700/186
• US Class Current: 382/153
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 30/24 involving temporal multiplexing, e.g. using sequentially activated left and right shutters
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

A telepresence system includes a camera pair remotely controlled by a control module affixed to an operator. The camera pair provides for three dimensional viewing and the control module, affixed to the operator, affords hands-free operation of the camera pair. In one embodiment, the control module is affixed to the head of the operator and an initial position is established. A triangulating device is provided to track the head movement of the operator relative to the initial position. A processor module receives input from the triangulating device to determine where the operator has moved relative to the initial position and moves the camera pair in response thereto. The movement of the camera pair is predetermined by a software map having a plurality of operation zones. Each zone therein corresponds to unique camera movement parameters such as speed of movement. Speed parameters include constant speed, or increasing or decreasing. Other parameters include pan, tilt, slide, raise or lowering of the cameras. Other user interface devices are provided to improve the three dimensional control capabilities of an operator in a local operating environment. Such other devices include a pair of visual display glasses, a microphone and a remote actuator. The pair of visual display glasses are provided to facilitate three dimensional viewing, hence depth perception. The microphone affords hands-free camera movement by utilizing voice commands. The actuator allows the operator to remotely control various robotic mechanisms in the remote operating environment.

Claims

1. A telepresence system for assisting in assimilating an operator in a local operating environment into a remote operating environment, the system comprising:

(a) a camera assembly comprising a pair of adjacently disposed cameras to obtain a visual image in three dimensions;

(b) means for orienting the camera assembly;

(c) a visual display unit configured to receive and display the visual image from the camera assembly;

(d) means for enabling the operator to view the visual image on the visual display unit in three dimensions;

(e) a control module configured for attachment to the operator;

(f) a triangulating device configured to receive signals from the control module; and

(g) a processor module configured to receive signals from the sensing unit and transmit signals to the means for orienting the camera such that the camera is oriented in response to the operators movement.

2. A telepresence system according to claim 1, wherein the means for orienting the camera assembly comprises a pan and tilt device.

3. A telepresence system according to claim 1, wherein the means for orienting the camera assembly comprises a sliding track.

4. A telepresence system according to claim 1, wherein the means for enabling the operator to view the visual image on the visual display unit in three dimensions comprises a pair of visual display glasses.

5. A telepresence system according to claim 4, wherein the visual display glasses have a right and left lens thereof and the visual image is cycled between the lenses.

6. A telepresence system according to claim 1, wherein the triangulating device comprises a plurality of microphones.

7. A telepresence system according to claim 1, further comprising:

(a) a robotic mechanism configured for operation in the remote operating environment; and

(b) an actuator configured to enable the user viewing the visual display to control the robotic mechanism.

8. A telepresence system according to claim 1, further comprising a pair of stereo microphones mounted to the camera assembly.

9. A telepresence system according to claim 1, further comprising:

(a) a plurality of camera pairs; and

(b) means for switching between activation of the camera paris by voice command.

10. A telepresence system for assisting in assimilating an operator in a local operating environment into a remote operating environment, the system comprising:

(a) a camera assembly configured to obtain a visual image in the remote operating environment;

(b) means for orienting the camera assembly using a select combination of movement parameters, the movement parameters including speed of camera movement and type of camera movement;

(c) a visual display unit configured to receive and display an interpretation of the visual image from the camera assembly;

(d) means for enabling tracking of the relative movement of the operator as the operator views the visual display; and

(e) a processor being configured to;

(i) initialize an initial position of the operator based on inputs received by the processor from the means for enabling tracking;

(ii) establish a map including a plurality of operation zones, the zones corresponding to regions in which the operator can move, the zones being positioned relative to the initial position of the operator;

(iii) receive inputs from the means for enabling tracking to ascertain the coordinates and corresponding zone of the operator as the operator moves while viewing the visual display; and

(iv) transmit signals to the means for orienting the camera, the signals defining the select combination of movement parameters for the camera, the select combination of movement parameters being unique for each of the plurality of operation zones.

11. A telepresence system according to claim 10, wherein the camera assembly comparises a pair of adjacently disposed cameras.

12. A telepresence system according to claim 10, wherein the means for orienting the camera assembly comprises a pan and tilt device.

13. A telepresence system according to claim 10, further comprising means for enabling the operator to view the visual image on the visual display unit in three dimensions.

14. A telepresence system according to claim 10, wherein the means for enabling tracking comprises:

(a) a control module configured for attachment to the head of the operator; and

(b) a triangulating device configured to receive signals from the control module.

15. A telepresence system according to claim 10, further comprising:

(a) a robotic mechanism configured for operation in the remote operating environment; and

(b) an actuator configured to enable the user viewing the visual display to control the robotic mechanism.

16. A telepresence system according to claim 10, wherein the plurality of operation zones comprises a first zone wherein the camera assembly remains stationary during movement of the operator in the region corresponding to the first zone.

17. A telepresence system according to claim 10, wherein the plurality of operation zones comprises a second zone wherein the camera assembly can pan and tilt at a first predetermined speed when the operated is in the region corresponding to the second zone.

18. A telepresence system according to claim 17, wherein the plurality of operation zones comprises a third zone wherein the camera assembly can pan and tilt at a second predetermined speed faster than the first predetermined speed when the operated is in the region corresponding to the third zone.

19. A telepresence system for assisting in assimilating an operator in a local operating environment into a remote operating environment, the system comprising:

(a) a camera pair configured to obtain a visual image in the remote operating environment;

(b) a pan and tilt device mounted to the camera pair;

(c) a sliding track having the camera mounted thereto, the sliding track being configured to move the camera pair along the length thereof;

(d) a visual display unit configured to receive and display the visual image from the camera pair;

(e) a control module configured for attachment to the operator;

(f) a triangulating device configured to receive signals from the control module;

(g) a processor being configured to;

(i) initialize an initial position of the operator based on inputs received by the processor from the triangulating device;

(ii) establish a map including a plurality of different zones, the zones corresponding to regions in which the operator can move, the zones including a first zone, a second zone, and a third zone, the zones being positioned relative to the initial position of the operator;

(iii) receive inputs from the triangulating device to ascertain the coordinates and corresponding zones of the operator as the operator moves while viewing the visual display unit;

(iv) transmit signals to the pan and tilt device for moving the camera along a path at a predetermined speed when the operator is in the second zone; and

(v) transmit signals to the sliding track for moving the camera therealong when the operator is in the third zone.

20. A telepresence system according to claim 19, further comprising means for enabling the operator to view the visual image on the visual display unit in three dimensions.

21. A telepresence system according to claim 19, further comprising:

(a) a robotic mechanism configured for operation in the remote operating environment; and

(b) an actuator configured to enable the user viewing the visual display to control the robotic mechanism.

22. A telepresence system according to claim 19, further comprising:

(a) a plurality of camera pairs; and

(b) means for switching between activation of the camera paris by voice command.

23. A telepresence system according to claim 19, wherein the camera pair remains stationary during movement of the operator in the region corresponding to the first zone.

24. A method for operating a processor as part of a telepresence system, the telepresence system further including a camera assembly configured to obtain a visual image, the camera being movable by a means for orienting the camera under a select combination of movement parameters, the movement parameters including speed of camera movement and type of camera movement, a visual display unit configured to receive and display the visual image from the camera, a control module configured for attachment to the operator, and a triangulating device configured to receive signals from the control module, the method comprising the steps of:

(a) initializing a coordinate system based on an initial position of the operator, the initial position being based on inputs received from the triangulating device;

(b) establishing a map including a plurality of different zones, the zones corresponding to regions in which the operator can move, the zones being positioned relative to the initial position of the operator;

(c) receiving inputs from the triangulating device to ascertain the coordinates and corresponding zone of the operator as the operator moves while viewing the visual display unit; and

(d) transmitting signals to the means for orienting the camera, the signals defining the select combination of movement parameters for the camera, the select combination of movement parameters being unique for each of the plurality of zones.

25. A method according to claim 24, further comprising the step of receiving voice command inputs to alter various system parameters.

26. A method according to claim 24, further comprising the step of receiving inputs from an actuator device for controlling movement of a robotic mechanism in a remote operating environment.

27. A method according to claim 24, wherein the step of transmitting signals to the means for orienting the camera comprises the step of transmitting a signal to slide the camera along the track.

28. A method according to claim 24, further comprising the step of transmitting an audio signal received from a remote operating environment to the operator in a local operating environment to facilitate assimilation of the operator into the remote operating environment.

29. A method according to claim 24, wherein the method steps are recorded on a computer readable medium.