System and method to display maintenance and operational instructions of an apparatus using augmented reality

• US 9,323,055 B2
• Filed: 05/26/2006
• Issued: 04/26/2016
• Est. Priority Date: 05/26/2006
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Exelis Incorporated (L3Harris Technologies, Inc.)
• Inventors: Baillot, Yohan
• Primary Examiner(s): MA, CALVIN
• Application Number:
  US11/441,241
• Publication Number:
  US 20070273610A1
• Time in Patent Office: 3,623 Days
• Field of Search: 345/7, 345/8, 356/620, 356/623
• US Class Current: 1/1
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft

Abstract

A head-mounted display provides a user with an augmented view of an object being viewed; a tracking mechanism such as a camera repeatedly determine the position and orientation of the head-mounted display relative the object being viewed; and a computer system provides information for the augmented view and repeatedly updates the augmented view of the object being viewed based on the determined position and orientation of the display. The head-mounted display may be a see-through display; it may be video-based or optical-based, and it may be monoscopic and stereoscopic. The tracking mechanism determines its position using one or more markers or beacons on the object being viewed. The markers may be active or passive, including light-emitting diodes (LEDs) that emit invisible light.

Claims

1. A framework comprising:

a head-mounted display constructed and adapted to provide a user with an augmented view of a physical object being viewed directly by the user, wherein the physical object is not a digital representation of an object and is one of;

(1) an apparatus, (2) a device, (3) an instrument panel and a (2) machine to be one of;

(1) maintained and (2) repaired;

a tracking mechanism constructed and adapted to repeatedly determine the position and orientation of the head-mounted display relative the object being viewed, wherein the tracking system is constructed and adapted to determine its position using a plurality of markers located on the object being viewed; and

a computer system constructed and adapted to provide one of (1) maintenance and (1) repair instructions displayed overlaid over the user'"'"'s direct view of the physical object at particular locations relative to the physical object being viewed to create the augmented view and to repeatedly update the augmented view of the physical object being viewed based, at least in part, on the determined position and orientation of the display.

2. A framework as in claim 1 wherein the head-mounted display is a see-through display.

3. A framework as in claim 2 wherein the head-mounted display is selected from:

video-based displays and optical-based displays.

4. A framework as in claim 1 wherein the head-mounted display is selected from:

monoscopic and stereoscopic displays.

5. A framework as in claim 1 wherein each of the plurality of markers comprises:

light-emitting diodes (LEDs).

6. A framework as in claim 5 wherein at least some of the LEDs are selected from infra-red (IR) LEDs and ultra-violet (UV) LEDs.

7. A framework as in claim 5 wherein the LEDs emit invisible light.

8. A framework as in claim 1 wherein the markers are retro-reflective targets, the framework further comprising an illumination mechanism.

9. A framework as in claim 1 wherein the tracking mechanism comprises a camera.

10. A framework as in claim 9 wherein the camera is connected to the head-mounted display.

11. A framework as in claim 9 wherein the camera uses an imaging sensor operating in a frequency range that is invisible to humans.

12. A framework as in claim 11 wherein the frequency is selected from infrared (IR) and ultraviolet (UV).

13. A framework as in claim 1 wherein the tracking mechanism includes one or more filters to filter out ambient illumination.

14. A framework as in claim 1 wherein the object being viewed is selected from a plurality of previously marked objects.

15. A framework as in claim 1 wherein the augmented view of the object presents the information on the object being viewed.

16. A framework as in claim 15 further comprising:

computer-readable media storing information associated with said plurality of previously-marked objects.

17. A framework as in claim 1 wherein the tracking mechanism is constructed and adapted to substantially continuously determine the position and orientation of the head-mounted display relative to the objected being viewed;

and wherein the computer system is constructed and adapted to substantially continuously update the augmented view of the object based on the determined position and orientation of the display.

18. A framework as in claim 1 wherein said instructions comprise condition information about a condition of the object and wherein the augmented display includes that condition information at an appropriate location on the object.

19. A framework as in claim 18 wherein the condition information comprises an internal temperature of the object.

20. A framework as in claim 1 wherein the tracking mechanisms uses an inside-out tracking system.

21. A method in a head-mounted display, comprising:

Providing a direct view of a physical object to the user, wherein the physical object is not a digital representation of an object and is one of;

(1) an apparatus, (2) a device, (3) an instrument panel and a (2) machine to be one of;

(1) maintained and (2) repaired;

repeatedly determining a relative position and orientation of the user with respect to the physical object being viewed by the user, wherein the physical object has a plurality of markers located thereon, and wherein said determining is based at least in part on at least some of the markers; and

based on said determined position and orientation, repeatedly providing the user with one of (1) maintenance and (1) repair instructions displayed overlaid over the user'"'"'s direct view of the physical object at particular locations relative to the physical object being viewed to create an augmented view of the physical object being viewed.

22. A method as in claim 21 wherein the steps are repeated substantially continuously.

23. A method as in claim 21 wherein the augmented view of the object provides the information on instructions for the object being viewed.

24. A method as in claim 23 wherein at least one of the particular locations is a location corresponding to a part of the object, and wherein the information is instructions are information relating to that part of the object.

25. A method, in a system having a plurality of actual objects, each of said plurality of actual objects having at least one distinct identification marker associated therewith and having a plurality of position markers associated therewith, the method comprising:

(a) obtaining an image, said image including a representation of an actual object, said image corresponding substantially with a user'"'"'s view of the actual object;

(b) identifying said actual object represented in said image, said identifying being based, at least in part, on at least one distinct identification marker associated with the actual object;

(c) determining a relative position and orientation of said user with respect to the identified actual object, said determining being based at least in part on at least some of said position markers; and

(d) providing the user with an augmented view of the actual object,

wherein the augmented view presents one of (1) maintenance and (1) repair instructions displayed overlaid over the user'"'"'s direct view of the object so as to appear at a specific location relative to a specific part of the actual object.

26. A method as in claim 25 further comprising:

repeating steps (a)-(d) at least once.