Optical see-through augmented reality modified-scale display

• US 7,002,551 B2
• Filed: 09/25/2002
• Issued: 02/21/2006
• Est. Priority Date: 09/25/2002
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Azuma, Ronald T., Sarfaty, Ron
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): GOOD JOHNSON, MOTILEWA
• Application Number:
  US10/256,090
• Publication Number:
  US 20040051680A1
• Time in Patent Office: 1,245 Days
• Field of Search: 345/633
• US Class Current: 345/158
• CPC Class Codes: G03B 13/28 Image-splitting devices
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method and system for providing an optical see-through Augmented Reality modified-scale display. This aspect includes a sensor suite 100 which includes a compass 102, an inertial measuring unit 104, and a video camera 106 for precise measurement of a user'"'"'s current orientation and angular rotation rate. A sensor fusion module 108 may be included to produce a unified estimate of the user'"'"'s angular rotation rate and current orientation to be provided to an orientation and rate estimate module 120. The orientation and rate estimate module 120 operates in a static or dynamic (prediction) mode. A render module 140 receives an orientation; and the render module 140 uses the orientation, a position from a position measuring system 142, and data from a database 144 to render graphic images of an object in their correct orientations and positions in an optical display 150.

Claims

1. An optical see-through imaging apparatus having variable magnification for producing an augmented image from a real scene and a computer generated image, the apparatus comprising:

a. a sensor suite for precise measurement of a user'"'"'s current orientation;

b. a render module connected with the sensor suite for receiving a sensor suite output comprising the user'"'"'s current orientation for use in producing the computer generated image of an object to combine with the real scene;

c. a position measuring system connected with the render module for providing a position estimation for producing the computer generated image of the object to combine with the real scene;

d. a database connected with the render module for providing data for producing the computer generated image of the object to combine with the real scene; and

e. an optical display connected with the render module configured to receive an optical view of the real scene, and for combining the optical view of the real scene with the computer generated image of the object from the render module to produce a display based on the user'"'"'s current position and orientation for a user to view.

2. An optical see-through imaging apparatus as set forth in claim 1 wherein:

a. the sensor suite further comprises an inertial measuring unit that includes at least one inertial angular rate sensor;

b. the apparatus further comprises a sensor fusion module connected with the inertial measuring unit for accepting an inertial measurement including a user'"'"'s angular rotation rate for use in determining a unified estimate of the user'"'"'s angular rotation rate and current orientation;

c. the render module is connected with the sensor fusion module for receiving a sensor fusion module output consisting of the unified estimate of the user'"'"'s angular rotation rate and current orientation from the sensor fusion module for use in producing the computer generated image of the object to combine with the real scene; and

d. the optical display further utilizes the unified estimate of the user'"'"'s angular rotation rate and current orientation from the sensor fusion module to produce the display based on the unified estimate of the user'"'"'s current position and orientation for a user to view.

3. An optical see-through imaging apparatus as set forth in claim 2 wherein:

a. the sensor suite further comprises a compass;

b. the sensor fusion module is connected with a sensor suite compass for accepting a sensor suite compass output from the sensor suite compass; and

c. the sensor fusion module further uses the sensor suite compass output in determining the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

4. An optical see-through imaging apparatus as set forth in claim 2 wherein:

a. the apparatus further comprises an orientation and rate estimator module connected with the sensor fusion module for accepting the sensor fusion module output consisting of the unified estimate of the user'"'"'s angular rotation rate and current orientation, and when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, the orientation and rate estimator module predicts a future orientation;

otherwise the orientation and rate estimator module uses the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. the render module is connected with the orientation and rate estimator module for receiving the predicted future orientation or the average orientation from the orientation and rate estimator module for use in producing the computer generated image of the object to combine with the real scene; and

c. the optical display is based on the predicted future orientation or the average orientation from the orientation and rate estimator module for the user to view.

5. An optical see-through imaging apparatus as set forth in claim 2 wherein:

a. the sensor suite further comprises a sensor suite video camera; and

b. the apparatus further comprises a video feature recognition and tracking movement module connected between the sensor suite video camera and the sensor fusion module, wherein the sensor suite video camera provides a sensor suite video camera output, including video images, to the video feature recognition and tracking movement module, and wherein the video feature recognition and tracking movement module provides a video feature recognition and tracking movement module output to the sensor fusion module, which utilizes the video feature recognition and tracking movement module output to provide increased accuracy in determining the unified estimate of the user'"'"'s angular rotation rate and current orientation.

6. An optical see-through imaging apparatus as set forth in claim 5 wherein:

a. the sensor suite further comprises a compass;

b. the sensor fusion module is connected with a sensor suite compass for accepting a sensor suite compass output from the sensor suite compass; and

c. the sensor fusion module further uses the sensor suite compass output in determining the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

7. An optical see-through imaging apparatus as set forth in claim 5 wherein:

a. the apparatus further comprises an orientation and rate estimator module connected with the sensor fusion module for accepting the sensor fusion module output consisting of the unified estimate of the user'"'"'s angular rotation rate and current orientation, and when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, the orientation and rate estimator module predicts a future orientation;

otherwise the orientation and rate estimator module uses the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. the render module is connected with the orientation and rate estimator module for receiving the predicted future orientation or the average orientation from the orientation and rate estimator module for use in producing the computer generated image of the object to combine with the real scene; and

c. the optical display is based on the predicted future orientation or the average orientation from the orientation and rate estimator module for the user to view.

8. An optical see-through imaging apparatus as set forth in claim 5, wherein the video feature recognition and tracking movement module further comprises a template matcher for more accurate registration of the video images for measuring the user'"'"'s current orientation.

9. An optical see-through imaging apparatus as set forth in claim 8 wherein:

a. the apparatus further comprises an orientation and rate estimator module connected with the sensor fusion module for accepting the sensor fusion module output consisting of the unified estimate of the user'"'"'s angular rotation rate and current orientation, and when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, the orientation and rate estimator module predicts a future orientation;

otherwise the orientation and rate estimator module uses the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. the render module is connected with the orientation and rate estimator module for receiving the predicted future orientation or the average orientation from the orientation and rate estimator module for use in producing the computer generated image of the object to combine with the real scene; and

c. the optical display is based on the predicted future orientation or the average orientation from the orientation and rate estimator module for the user to view.

10. An optical see-through imaging apparatus as set forth in claim 8 wherein:

a. the sensor suite further comprises a compass;

b. the sensor fusion module is connected with a sensor suite compass for accepting a sensor suite compass output from the sensor suite compass; and

c. the sensor fusion module further uses the sensor suite compass output in determining the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

11. An optical see-through imaging apparatus as set forth in claim 10 wherein:

a. the apparatus further comprises an orientation and rate estimator module connected with the sensor fusion module for accepting the sensor fusion module output consisting of the unified estimate of the user'"'"'s angular rotation rate and current orientation, and when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, the orientation and rate estimator module predicts a future orientation;

otherwise the orientation and rate estimator module uses the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. the render module is connected with the orientation and rate estimator module for receiving the predicted future orientation or the average orientation from the orientation and rate estimator module for use in producing the computer generated image of the object to combine with the real scene; and

c. the optical display is based on the predicted future orientation or the average orientation from the orientation and rate estimator module for the user to view.

12. A method for optical see-through imaging through an optical display having variable magnification for producing an augmented image from a real scene and a computer generated image, the method comprising the steps of:

a. measuring a user'"'"'s current orientation by a sensor suite;

b. rendering the computer generated image by combining a sensor suite output connected with a render module, a position estimation output from a position measuring system connected with the render module, and a data output from a database connected with the render module;

c. displaying the combined optical view of the real scene and the computer generated image of an object in the user'"'"'s current position and orientation for the user to view through the optical display connected with the render module; and

d. repeating the measuring step through the displaying step to provide a continual update of the augmented image.

13. A method for optical see-through imaging through an optical display set forth in claim 12, further comprising the step of:

a. producing a unified estimate of a user'"'"'s angular rotation rate and current orientation from a sensor fusion module connected with the sensor suite, wherein the sensor suite includes an inertial measuring unit that includes at least one inertial angular rate sensor for measuring the user'"'"'s angular rotation rate;

b. wherein the rendering the computer generated image step includes a unified estimate of the user'"'"'s angular rotation rate and current orientation from the sensor fusion module; and

c. wherein the displaying the combined optical view step includes the unified estimate of the user'"'"'s angular rotation rate and current orientation.

14. A method for optical see-through imaging through an optical display set forth in claim 13, wherein the step of measuring the user'"'"'s current orientation by a sensor suite includes measuring the user'"'"'s current orientation using a compass, and wherein the measurements produce the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

15. A method for optical see-through imaging through an optical display set forth in claim 13:

a. further comprising the step of predicting a future orientation at the time a user will view a combined optical view by an orientation and rate estimate module connected with and using output from the sensor fusion module when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, otherwise using the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. wherein the rendering the computer generated image step may include a predicted future orientation output from the orientation and rate estimate module; and

c. wherein the displaying the combined optical view step may include a predicted future orientation.

16. A method for optical see-through imaging through an optical display set forth in claim 13, wherein the step of measuring the user'"'"'s orientation by a sensor suite further includes measuring the user'"'"'s orientation using a video camera, and a video feature recognition and tracking movement module wherein the video feature recognition and tracking movement module receives a sensor suite video camera output from a sensor suite video camera and provides the sensor fusion module measurements to enable the sensor fusion module to produce the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

17. A method for optical see-through imaging through an optical display set forth in claim 16, wherein the step of measuring the user'"'"'s current orientation by a sensor suite includes measuring the user'"'"'s current orientation using a compass, and wherein the measurements produce the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

18. A method for optical see-through imaging through an optical display set forth in claim 16:

a. further comprising the step of predicting a future orientation at the time a user will view a combined optical view by an orientation and rate estimate module connected with and using output from the sensor fusion module when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, otherwise using the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. wherein the rendering the computer generated image step may include a predicted future orientation output from the orientation and rate estimate module; and

c. wherein the displaying the combined optical view step may include a predicted future orientation.

19. A method for optical see-through imaging through an optical display set forth in claim 16, wherein the step of measuring the user'"'"'s orientation further includes a template matcher within the video feature recognition and tracking movement module, and provides the sensor fusion module measurements to enable the sensor fusion module to produce the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

20. A method for optical see-through imaging through an optical display set forth in claim 19:

a. further comprising the step of predicting a future orientation at the time a user will view a combined optical view by an orientation and rate estimate module connected with and using output from the sensor fusion module when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, otherwise using the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. wherein the rendering the computer generated image step may include a predicted future orientation output from the orientation and rate estimate module; and

c. wherein the displaying the combined optical view step may include a predicted future orientation.

21. A method for optical see-through imaging through an optical display set forth in claim 19, wherein the step of measuring the user'"'"'s current orientation by a sensor suite includes measuring the user'"'"'s current orientation using a compass, and wherein the measurements produce the unified estimate of the user'"'"'s angular rotation rate and current orientation with increased accuracy.

22. A method for optical see-through imaging through an optical display set forth in claim 21:

a. further comprising the step of predicting a future orientation at the time a user will view a combined optical view by an orientation and rate estimate module connected with and using output from the sensor fusion module when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, otherwise using the unified estimate of the user'"'"'s current orientation to produce an average orientation;

b. wherein the rendering the computer generated image step may include a predicted future orientation output from the orientation and rate estimate module; and

c. wherein the displaying the combined optical view step may include a predicted future orientation.

23. An orientation and rate estimator module for use with an optical see-through imaging apparatus, the module comprising:

a. a means for accepting a sensor fusion modular output consisting of a unified estimate of a user'"'"'s angular rotation rate and current orientation;

b. a means for using the sensor fusion modular output to generate a future orientation when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, otherwise the orientation and rate estimator module generates a unified estimate of the user'"'"'s current orientation to produce an average orientation; and

c. a means for outputting the future orientation or the average orientation from the orientation and rate estimator module for use in the optical see-through imaging apparatus for producing a display based on the unified estimate of the user'"'"'s angular rotation rate and current orientation.

24. An orientation and rate estimator module as set forth in claim 23, wherein the orientation and rate estimator module is configured to receive a sensor fusion module output wherein the sensor fusion module output includes data selected from the group consisting of an inertial measuring unit output, a compass output, and a video camera output.

25. A method for orientation and rate estimating for use with an optical see-through image apparatus, the method comprising the steps of:

a. accepting a sensor fusion modular output consisting of the unified estimate of the user'"'"'s angular rotation rate and current orientation;

b. using the sensor fusion modular output to generate a future orientation when the user'"'"'s angular rotation rate is determined to be above a pre-determined threshold, otherwise generating a unified estimate of the user'"'"'s current orientation to produce an average orientation; and

c. outputting the future orientation or the average orientation from the orientation and rate estimator module for use in the optical see-through imaging apparatus for producing a display based on the unified estimate of the user'"'"'s angular rotation rate and current orientation.