Augmented reality aircraft management system

• US 9,019,128 B1
• Filed: 05/21/2013
• Issued: 04/28/2015
• Est. Priority Date: 05/21/2013
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Kim, Guen Il
• Primary Examiner(s): Swarthout, Brent
• Application Number:
  US13/898,686
• Time in Patent Office: 707 Days
• Field of Search: 340/945, 340/971, 340/973, 340/974, 340/975, 701/3, 701/14
• US Class Current: 340/945
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration

Abstract

A method and apparatus for providing assistance to a flight crew during flight. An image of a group of instruments on a flight deck of an aircraft is generated with an image sensor in a data processing system. Assistance information configured to assist the flight crew from flight information displayed by the group of instruments in the image is generated. The assistance information is displayed on a display device in the data processing system.

Claims

1. A method for providing assistance to a flight crew during flight, the method comprising:

receiving, in an information identifier component of a computer system, an image of flight information from an image sensor;

processing, in the information identifier, the image and generating graphical indicators, for a group of instruments on a flight deck of an aircraft, on a display device;

determining, using a data processor in a flight assistance system, whether the information identifier requires a user input to identify assistance information for an instrument in the group of instruments, and, when the information identifier requires the user input, displaying, on the display device, a group of actions to be performed;

generating, in the information identifier, using the flight information displayed by the group of instruments in the image, assistance information that assists the flight crew; and

displaying the assistance information on the display device.

2. The method of claim 1, wherein displaying the assistance information on the display device in the data processing system comprises:

displaying a live view of the group of instruments on the display device; and

displaying the assistance information on the live view of the group of instruments.

3. The method of claim 2, wherein the live view and the assistance information are for an augmented reality view displayed on the display device.

4. The method of claim 1, wherein the group of actions generate additional information needed for an analysis to generate the assistance information for the instrument selected from the group of instruments.

5. The method of claim 1, wherein the group of actions is selected from at least one of:

generating an image of a group of parameters displayed on another group of instruments, and changing a display in the group of instruments.

6. The method of claim 1, wherein generating the assistance information comprises:

searching a group of sources for the assistance information.

7. The method of claim 6, wherein the group of sources is selected from at least one of a local database on the aircraft, a remote database located off the aircraft, a web service, and a cloud computing system.

8. The method of claim 1, wherein generating the assistance information comprises:

displaying a group of graphical indicators in association with the group of instruments in the image, wherein the group of graphical indicators indicates that the group of instruments is selectable; and

identifying the assistance information from the flight information for the instrument in the group of instruments.

9. The method of claim 8, wherein the flight information is at least one of a parameter for the flight of the aircraft or a message.

10. The method of claim 1, wherein the assistance information comprises at least one of a description, an analysis, a possible solution, or company specific information.

19. The method of claim 1, wherein generating the image comprises a static image of the flight information generated at a time prior to generating the assistance information.

20. The method of claim 1, wherein displaying the assistance information comprises removing the image of the group of instruments from the display device.

11. An apparatus, comprising a computer system comprising an information identifier configured to:

receive from an image sensor, an image of a group of instruments in a flight deck of an aircraft;

determine whether the information identifier requires a user input to identify assistance information for an instrument in the group of instruments, such that when the information identifier requires the user input, a display device displays a group of actions to be performed;

generate, using flight information displayed by the group of instruments in the image, assistance information configured to assist a flight crew; and

display the assistance information on the display device, such that, in operation, the information identifier;

receives from the image sensor, the image of the group of instruments in the flight deck of the aircraft;

determines whether the information identifier requires the user input to identify the assistance information for the instrument in the group of instruments, such that when the information identifier requires the user input, the display device displays the group of actions to be performed;

generates, using the flight information displayed by the group of instruments in the image, the assistance information that assists the flight crew.

12. The apparatus of claim 11, wherein the information identifier displays a live view of the group of instruments on the display device and display the assistance information on the live view of the group of instruments.

13. The apparatus of claim 12, wherein the live view and the assistance information are for an augmented reality view displayed on the display device.

14. The apparatus of claim 11, wherein the group of actions generate additional information needed for an analysis to generate the assistance information for the instrument selected from the group of instruments.

15. The apparatus of claim 11, wherein the group of actions is selected from at least one of:

generating the image of a group of parameters displayed on another group of instruments, and

changing a display in the group of instruments.

16. The apparatus of claim 11, wherein the information identifier identifies a question from the image of the group of instruments and search a group of sources for the assistance information for the question.

17. A flight assistance system comprising:

a data processing system;

a display device in the data processing system; and

a computer system comprising an information identifier configured to generate;

an image, using data from an image sensor associated with the data processing system, of a group of instruments in a flight deck of an aircraft;

a determination of whether the information identifier requires a user input to identify assistance information for an instrument in the group of instruments, and, when the information identifier requires the user input, display, on the display device, a group of actions to be performed;

the assistance information, using flight information displayed by the group of instruments in the image, configured to assist a flight crew; and

a display, of the assistance information, on the display device in the data processing system, such that, in operation, the information identifier generates;

the image, using data from the image sensor associated with the data processing system, of the group of instruments in the flight deck of the aircraft;

the determination of whether the information identifier requires the user input to identify assistance information for the instrument in the group of instruments, and, when the information identifier requires the user input, display, on the display device, the group of actions to be performed;

the assistance information, using flight information displayed by the group of instruments in the image, that assists the flight crew; and

the display, of the assistance information, on the display device in the data processing system.

18. The flight assistance system of claim 17, wherein the information identifier displays a live view of the group of instruments on the display device and display the assistance information on the live view of the group of instruments.