System and method to display maintenance and operational instructions of an apparatus using augmented reality

• US 9,324,229 B2
• Filed: 01/19/2012
• Issued: 04/26/2016
• Est. Priority Date: 03/08/2007
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Exelis Incorporated (L3Harris Technologies, Inc.)
• Inventors: Baillot, Yohan
• Primary Examiner(s): MA, CALVIN
• Application Number:
  US13/354,167
• Publication Number:
  US 20120120070A1
• Time in Patent Office: 1,559 Days
• Field of Search: 345 7- 8, 345/419, 345/633, 356/620, 356/623
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G08B 13/19621 Portable camera
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons
• CPC Class Codes: G08B 25/016 Personal emergency signalling and security systems emergency non-personal manually actuated alarm activators G08B25/12
• CPC Class Codes: G08B 25/14 Central alarm receiver or annunciator arrangements

Abstract

A head-mounted display provides a user with an augmented view of an object being viewed; a tracking mechanism such as a camera repeatedly determine the position and orientation of the head-mounted display relative the object being viewed; and a computer system provides information for the augmented view and repeatedly updates the augmented view of the object being viewed based on the determined position and orientation of the display. The head-mounted display may be a see-through display; it may be video-based or optical-based, and it may be monoscopic and stereoscopic. The tracking mechanism determines its position using one or more markers or beacons on the object being viewed. The markers may be active or passive, including light-emitting diodes (LEDs) that emit invisible light.

Claims

1. An augmented-reality display system for providing repair and maintenance information, comprising:

a head-mounted display configured to;

display an augmented view of a physical device to be one of;

(1) repaired and (2) maintained, by presenting one of;

(1) computer-generated repair information about the physical device and (2) computer-generated maintenance information about the physical device in an image superimposed over a real-time view of the physical device, wherein a user views the physical device through the head-mounted display; and

determine a position and orientation of a tracking device relative to active markers on the physical device; and

repeatedly update the superimposed information image as the head-mounted display and physical device move relative to each other using the determined position and orientation of the tracking device relatively to the active markers; and

the tracking device configured to locate the active markers on the physical device.

2. The augmented-reality display system of claim 1 wherein the head-mounted display is a see-through display.

3. The augmented-reality display system of claim 2 wherein the head-mounted display is an optical-based display.

4. The augmented-reality display system of claim 1 wherein the head-mounted display is selected from:

monoscopic and stereoscopic displays.

5. The augmented-reality display system of claim 1 wherein the active markers comprise:

light-emitting diodes (LEDs).

6. The augmented-reality display system of claim 5 wherein at least some of the LEDs are selected from infra-red (IR) LEDs and ultra-violet (UV) LEDs.

7. The augmented-reality display system of claim 5 wherein the LEDs emit invisible light.

8. The augmented-reality display system of claim 1 wherein the tracking device comprises a camera.

9. The augmented-reality display system of claim 8 wherein the camera is connected to the head-mounted display.

10. The augmented-reality display system of claim 8 wherein the camera uses an imaging sensor operating in a frequency range that is invisible to humans.

11. The augmented-reality display system of claim 10 wherein the frequency is selected from infrared (IR) and ultraviolet (UV).

12. The augmented-reality display system of claim 1 wherein the tracking device includes one or more filters to filter out ambient illumination.

13. The augmented-reality display system of claim 1 wherein the physical device being viewed is selected from a plurality of previously marked physical devices.

14. The augmented-reality display system of claim 13 further comprising:

computer-readable media storing information associated with the plurality of previously-marked physical devices.

15. The augmented-reality display system of claim 14 wherein the information comprises condition information about a condition of the physical device and wherein the superimposed image includes that condition information at an appropriate location on the physical device.

16. The augmented-reality display system of claim 15 wherein the condition information comprises an internal temperature of the object.

17. A method in an augmented-reality display system for providing repair and maintenance information, comprising:

displaying an augmented view of a physical device to be one of (1) repaired and (1) maintained, by presenting one of;

(1) computer-generated repair information about the physical device and (2) computer-generated maintenance information about the physical device superimposed in an image over a real-time view of the physical device, wherein a user views the physical device through the head-mounted display;

locating the active markers on the physical device;

determining a position and orientation of a tracking device relative to the active markers on the physical device; and

repeatedly updating the superimposed repair or maintenance information image as the head-mounted display and physical device move relative to each other using the determined positioning and orientation of the tracking device relatively to the active markers.

18. A method as in claim 17 wherein the steps are repeated substantially continuously.