Method and system for time/motion compensation for head mounted displays

• US 7,312,766 B1
• Filed: 09/22/2000
• Issued: 12/25/2007
• Est. Priority Date: 09/22/2000
• Status: Expired due to Fees

• Current Assignee: Canadian Space Agency (Government of Canada)
• Original Assignee: Canadian Space Agency (Government of Canada)
• Inventors: Edwards, Eric C.
• Primary Examiner(s): PIZIALI, JEFFREY J
• Application Number:
  US10/088,747
• Time in Patent Office: 2,650 Days
• Field of Search: 345/1.1, 345/1.3, 345 7- 9, 345/156, 345/158, 345/162, 345/184, 345/427, 345/474, 345/592, 345/595, 345/620, 345/649, 340901-905, 340/980, 359/13, 359630-633, 348/42, 348/47, 348/94, 348 50- 53, 348/115, 348/117, 348/123, 348/129, 348/130, 348/135, 348/139, 348142-144, 348/148, 348/161, 348/169, 348/207.11, 348/208.2, 348/208.4, 348/208.14, 348/211.7, 348/211.8, 348/211.9, 348/211.11, 348/211.14, 348/376
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A method and system for time-motion compensation for use with head mounted displays is disclosed. According to the method, a remote camera captures an image for display on a head-mounted display (HMD) including a viewing window. The image and camera position data is transmitted to a system including the HMD for display to a wearer of the HMD. The HMD position is determined. An offset between the HMD position and a known position of the HMD is determined as is an offset between the camera position and a known position of the camera. The image is offset relative to the viewing window of the image based on the difference between the two determined offsets.

Claims

1. A method of motion compensation for a head mounted display (HMD) wherein a camera tracks movement, subject to possible delay, of an operator'"'"'s head to capture images within a field of view controlled by the orientation of the operator'"'"'s head, comprising the steps of:

sending a sequence of captured images at a video capture rate from the camera to a processor associated with the head mounted display;

obtaining camera position data identifying the field of view of the camera for each captured image;

associating said camera position data with the corresponding captured images;

providing HMD position data corresponding to a current position of the head mounted display;

for each captured image received by the computer comparing the HMD position data with the camera position data associated with the captured image to determine whether an offset exists between a first field of view associated with the captured image and a second field of view corresponding to the current position of the head mounted display;

in the absence of a said offset displaying the captured image or a portion thereof in the head mounted display; and

in the presence of a said offset, transforming the captured image to create a transformed image comprising a portion of said captured image that overlaps said first and second fields of view and displaying said transformed image in the head mounted display in a part of said second field of view, the position of said part of said second field of view in the head mounted display corresponding to the position of the overlapping portion of the captured image in the second field of view.

2. A method according to claim 1, wherein a remaining part of said second field of view not displaying said transformed image is filled with a predetermined fill.

3. A method according to claim 2, wherein the predetermined fill has features to assist the operator in maintaining personal orientation.

4. A method according to claim 1, wherein the captured image is larger than the field of view of the head mounted display, and only a portion of the captured image is displayed therein.

5. A method according to claim 1, wherein a first independent position sensor senses the camera position and provides the camera position data.

6. A method according to claim 5, wherein a second independent position sensor senses the head mounted display position and provides the HMD position data.

7. A method according to claim 1, wherein an independent position sensor senses the head mounted display position and provides the HMD position data.

8. A motion compensation apparatus according to claim 6, further comprising:

a mechanism for moving the camera; and

a communications link for transmitting the HMD position data to a system in controlling the mechanism for moving the camera.

9. A method according to claim 1, wherein the HMD position data comprises orientation data.

10. A method according to claim 9, wherein the camera position data comprises orientation data.

11. A method according to claim 10, wherein the camera position data comprises displacement data.

12. A method according to claim 1, wherein the HMD position data comprises displacement data.

13. A method according to claim 12, wherein the camera position data comprises displacement data.

14. A method according to claim 13, wherein the camera position data comprises orientation data.

15. A method according to claim 1, wherein the camera includes a range sensor to capture depth data, and the captured images are transformed using said depth data to reduce perspective distortion.

19. A method according to claim 1 wherein, when parts of the second field of view for which image data is unavailable are detected, earlier captured image data is located, which when transformed in dependence upon its associated camera position data and the HMD position data is for display within said parts;

and,

the transformed earlier captured image data is displayed in those parts of the field of view.

20. A method according to claim 19, wherein the transformed earlier captured image data is de-empasized relative to current captured image data.

16. A motion compensation apparatus for a head mounted display (HMD) wherein camera tracks movement subject to possible delay, of an operator'"'"'s head to capture images within a field of view controlled by the orientation of the operator'"'"'s head, comprising:

said head mounted display including a monitor having a field of view for displaying captured images;

a first processor associated with the head mounted display for receiving captured images for display in said head mounted display;

said camera for sending a sequence of captured images at a video capture rate to said first processor;

a first sensor for providing camera position data relating to a position of the camera for each captured image;

a second processor for associating the camera position data with the corresponding images sent to the first processor;

a second sensor for providing HMD position data relating to a position of the head mounted display; and

said first processor, for each captured image received thereby, comparing the HMD position data with the camera position data associated with the captured image to determine whether an offset exists between a first field of view associated with the captured image and a second field of view corresponding to the current position of the head mounted display; and

in the absence of a said offset said first processor displaying the captured image or a portion thereof in the head mounted display; and

in the presence of a said offset, said first processor transforming the captured image to create a transformed image comprising a portion of said captured image that overlaps said first and second fields of view and displaying said transformed image in the head mounted display in a part of said second field of view, the position of said part of said second field of view in the head mounted display corresponding to the position of the overlapping portion of the captured image in the second field of view.

17. A motion compensation apparatus according to claim 16, wherein the camera includes a range sensor.

18. A motion compensation apparatus according to claim 16, wherein the camera forms part of a three-dimensional vision system.