System and method for monitoring a cockpit of an aircraft

• US 10,252,815 B2
• Filed: 04/28/2017
• Issued: 04/09/2019
• Est. Priority Date: 04/28/2017
• Status: Active Grant

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Murugappan, Sundar, Carroll, Alexander Kaber, Ovens, Norman Leonard, Green, Sharon Ann, Cooper, Jennifer Ruth, Kim, So Young, Figard, Michael Eric, Akiyama, Masaki Merritt, Pires, Bernardo
• Primary Examiner(s): Patel, Shardul D
• Application Number:
  US15/581,016
• Publication Number:
  US 20180312272A1
• Time in Patent Office: 711 Days
• Field of Search: 701 305, 348143, 348160, 348 77, 340945, 340963, 340971
• CPC Class Codes: B64C 13/04 actuated personally
• CPC Class Codes: B64D 2045/0085 Devices for aircraft health monitoring, e.g. monitoring flutter or vibration
• CPC Class Codes: B64D 43/00 Arrangements or adaptations of instruments arrangements of cameras B64D47/08; aeronautical measuring instruments per se G01C
• CPC Class Codes: B64D 45/00 Aircraft indicators or protectors not otherwise provided for camouflage F41H3/00
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method of monitoring a cockpit of an aircraft includes receiving, by one or more controllers, an image depicting an operator manipulated input device located within the cockpit. The method can include determining, by the one or more controllers, an observed state of the operator manipulated input device. In particular, the observed state can be based on the image. The method can include determining, by the one or more controllers, a sensed state of the operator manipulated input device. In particular, the sensed state can be based on data from a sensor. The method can include determining, by the one or more controllers, a mismatch between the observed and sensed states of the operator manipulated input device.

Claims

1. A method of monitoring a cockpit of an aircraft, the method comprising:

receiving, by one or more controllers, an image depicting an operator manipulated input device located within the cockpit;

determining, by the one or more controllers, an observed state of the operator manipulated input device based on the image;

determining, by the one or more controllers, a sensed state of the operator manipulated input device based on data from a sensor;

determining, by the one or more controllers, a mismatch between the observed and sensed states of the operator manipulated input device;

wherein the operator manipulated input device comprises a throttle input device operable to adjust engine power of a gas turbine engine of the aircraft; and

wherein determining the observed state of the operator manipulated input device comprises;

converting, by the one or more controllers, the image to a commission on illumination (CIE) Lab color space; and

comparing the one or more controllers, the converted image against at least a first reference image and a second reference image, wherein the first reference image depicts the operator manipulated input device at a first position, and wherein the second reference image depicts the operator manipulated input device at a second position.

2. The method of claim 1, further comprising generating, by the one or more controllers, a notification indicating the mismatch, the notification comprising an audible or visual notification.

3. The method of claim 1, wherein the first reference image defines a region indicating location of the operator manipulated input device when in the first position, and wherein the second reference image defines a region indicating location of the operator manipulated input device when in the second position.

4. The method of claim 3, wherein the region defined by the first reference image encompasses a first set of pixels, wherein the region defined by the second reference image encompasses a second set of pixels, and wherein the first set of pixels is different than the second set of pixels.

5. The method of claim 1, wherein determining the observed state of the operator manipulated input device comprises:

converting, by the one or more controllers, the image to a grayscale image;

applying, by the one or more controllers, a threshold to each pixel of the grayscale image to obtain a binary image; and

comparing, by the one or more controllers, the binary image against at least a first reference image and a second reference image.

6. The method of claim 5, wherein the binary image, first reference image, and second reference image each depict a pattern that identifies the operator manipulated input device.

7. A method of monitoring a cockpit of an aircraft, the method comprising:

receiving, by one or more controllers, an image depicting an instrument within the cockpit;

determining, by the one or more controllers, an observed state of the instrument based on the image;

determining, by the one or more controllers, a sensed state of the instrument based on data from a sensor;

determining, by the one or more controllers, a mismatch between the observed and sensed states of the instrument; and

adjusting operation of the aircraft to correct for the mismatch;

wherein the instrument comprises a gauge having a needle that is movable between a first position and a second position.

8. The method of claim 7, further comprising generating, by the one or more controllers, a notification indicating the mismatch, the notification comprising an audible or visual notification.

9. The method of claim 7, wherein determining the observed state of the instrument comprises determining, by the one or more controllers, a position of the needle depicted in the image.

10. The method of claim 9, wherein determining the position of the needle comprises comparing, by the one or more controllers, the position of the needle in the image against at least a first reference image and a second reference image, the first reference image depicting the needle in the first position, the second reference image depicting the needle in the second position.

11. A system for monitoring a cockpit of an aircraft, the system comprising:

a camera positioned within the cockpit;

a flight deck disposed within the cockpit, the flight deck comprising a flight display; and

a controller in communication with the camera, the controller configured to;

receive an image from the camera, the image depicting the flight display;

determine an observed state of the flight display based on the image;

determine a sensed state of the flight display based on data from a sensor; and

determine a mismatch between the observed and sensed states of the flight display;

wherein the controller determines the observed state of the flight display by;

converting the image to a commission on illumination (CIE) Lab color space; and

comparing the converted image against at least a first reference image and a second reference image, wherein the first reference image depicts the operator manipulated input device at a first position, and wherein the second reference image depicts the operator manipulated input device aa second position.

12. The system of claim 11, wherein the controller is further configured to generate a notification indicating mismatch, the notification comprising an audible or visual notification.

13. The, system of claim 11, wherein the system further comprises an operator manipulated input device positioned within the cockpit, wherein the image further depicts the operator manipulated input device, and wherein the controller is further configured to:

determine an observed state of the operator manipulated input device based on the image;

determine a sensed state of the operator manipulated input device based on data from the sensor associated with the aircraft; and

determine a mismatch between the observed and sensed states of the operator manipulated input device.

14. The system of claim 13, wherein the flight deck further comprises an instrument, wherein the image further depicts the instrument, and wherein the controller is further configured to:

determine an observed state of the instrument based on the image;

determine a sensed state of the instrument based on data from a sensor associated with the operator manipulated input device; and

determine a mismatch between the observed and sensed states of the instrument.

15. The system of claim 11, wherein the observed state of the flight display corresponds to a numerical value displayed on the flight display.

16. The system of claim 11, wherein the controller is further configured to populate the flight display with information from another flight display when the mismatch exists between the observed and sensed states of the flight display.