System for optical recognition, interpretation, and digitization of human readable instruments, annunciators, and controls

• US 9,202,098 B2
• Filed: 08/15/2008
• Issued: 12/01/2015
• Est. Priority Date: 08/17/2007
• Status: Active Grant

• Current Assignee: Textron Innovations Incorporated (Textron Inc)
• Original Assignee: Textron Innovations Incorporated (Textron Inc)
• Inventors: Lewis, George Steven, Clanton, Carson Blaine
• Primary Examiner(s): Kim, Hee-Yong
• Application Number:
  US12/673,564
• Publication Number:
  US 20110149067A1
• Time in Patent Office: 2,664 Days
• Field of Search: 348/135, 375/240.08
• US Class Current: 1/1
• CPC Class Codes: B64D 45/00 Aircraft indicators or protectors not otherwise provided for camouflage F41H3/00
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G01D 2207/10 Displays which are primarily used in aircraft or display aircraft-specific information
• CPC Class Codes: G01D 3/08 with provision for safeguarding the apparatus, e.g. against abnormal operation, against breakdown
• CPC Class Codes: G01D 7/02 Indicating value of two or more variables simultaneously
• CPC Class Codes: G01P 1/08 Arrangements of scales, pointers, lamps or acoustic indicators, e.g. in automobile speedometers
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 2201/02 Recognising information on displays, dials, clocks
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera

Abstract

A system for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls includes an image acquisition sensor operable to capture images of at least one of the instruments, annunciators, and controls; and a logic processing unit operable to decode the images captured by the image acquisition sensor and interpret the decoded images to determine a state of the at least one of the instruments, annunciators, and controls.

Claims

1. A system for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls, comprising:

an image acquisition sensor operable to capture images of multiple instruments, annunciators, and controls; and

a logic processing unit operable to decode the images captured by the image acquisition sensor and interpret the decoded images to determine a state of the multiple instruments, annunciators, and controls;

wherein the human readable instruments, annunciators, and controls are objects in an aircraft cockpit;

wherein the system detects when at least one of the objects is occluded from the images captured by the image acquisition sensor;

wherein the system groups the objects into an occluded group of objects and a non-occluded group of objects; and

wherein the system continues to determine the state of the non-occluded group of objects while the at least one of the multiple instruments, annunciators, and controls is occluded.

2. The system of claim 1, wherein the logic processing unit is operable to store the state of at least one of the multiple instruments, annunciators, and controls.

3. The system of claim 1, wherein the logic processing unit is operable to store time-based states of at least one of the multiple instruments, annunciators, and controls.

4. The system of claim 1, wherein the logic processing unit creates a pseudo-parameter from the state of at least one of the multiple instruments, annunciators, and controls.

5. The system of claim 1, further comprising:

a ground data processor operably associated with the logic processing unit.

6. The system of claim 1, wherein the aircraft is a rotorcraft.

7. A method for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls, comprising:

reading a configuration file to provide a definition of a target region of an image to be interpreted of at least one of the instruments, annunciators, and controls to be interpreted;

opening an output file for storing the image to be interpreted;

determining whether a captured image to be interpreted exists;

if the captured image exists;

decoding the captured image;

determining a state of the at least one of the instruments, annunciators, and controls based upon the decoded captured image and the definition in the configuration file;

detecting when the at least one of the instruments, annunciators, and controls are not occluded from the captured image;

ignoring the state of the at least one of the instruments, annunciators, and controls that are occluded; and

writing the state of the at least one of the non-occluded instruments, annunciators, and controls to the output file.

8. The method of claim 7, wherein the configuration file includes one or more of:

a number of the at least one of the instruments, annunciators, and controls to be interpreted;

types of the at least one of the instruments, annunciators, and controls to be interpreted;

locations of the at least one of the instruments, annunciators, and controls to be interpreted;

a definition of a scan region of the at least one of the instruments, annunciators, and controls to be interpreted;

value maps associated with the locations of the at least one of the instruments, annunciators, and controls to be interpreted;

a mathematical representation of a reference region of the image for use in image registration;

a mathematical representation of reference patterns used to determine states of the at least one of the instruments, annunciators, and controls to be interpreted;

chromacity and intensity information for use in interpreting illumination status for an annunciator;

chromacity and intensity information for use in interpreting background versus foreground information of the image; and

startup parameters and preferences.

9. The method of claim 7, wherein the output file includes a time history of parameters interpreted and analyzed based upon definitions read from the configuration file.

10. The method of claim 7, wherein decoding the image results in a two-dimensional pixel bitmap or raster stored as a two-dimensional array.

11. The method of claim 7, wherein determining the state of the at least one of the instruments, annunciators, and controls is accomplished by:

registering the captured image with the definition of the target region of the image to be interpreted;

adjusting coordinates of the target region defined by the configuration file based upon registering the captured image;

scanning each pixel of the registered, captured image;

applying a median filter to each pixel of the registered, captured image;

segmenting the target region into foreground and background; and

picking foreground pixels corresponding to a variable feature of the at least one of the instruments, annunciators, and controls to be interpreted.

12. The method of claim 11, wherein registering the captured image is accomplished by:

computing a Fourier transform of the captured image;

element-wise multiplying a reference Fourier transform conjugate from the configuration file with the Fourier transform of the captured image;

computing an inverse Fourier transform of the multiplied reference Fourier transform conjugate and the Fourier transform of the captured image;

finding row and column indices of a maximum value in an array resulting from the inverse Fourier transform; and

storing the row and column indices.

13. The method of claim 12, further comprising:

element-wise normalizing the multiplied reference Fourier transform conjugate and the Fourier transform of the captured image prior to computing the inverse Fourier transform.

14. The method of claim 11, wherein segmenting the target region into foreground and background is accomplished by:

computing a threshold value for the target region based upon results of applying the median filter to each pixel of the registered, captured image;

retrieving an intensity of each pixel in the target area of the captured image;

comparing the intensity of each pixel to the threshold value; and

classifying each pixel as foreground or background.

15. The method of claim 11, wherein picking foreground pixels corresponding to a variable feature of the at least one of the instruments, annunciators, and controls to be interpreted is accomplished by:

windowing the captured image around the target area;

registering the windowed area of the captured image with the definition of the target region of the image to be interpreted; and

determining the state of the at least one of the instruments, annunciators, and controls to be interpreted based on the result of registering the windowed area.