System for Optical Recognition, Interpretation, and Digitization of Human Readable Instruments, Annunciators, and Controls

• US 20110149067A1
• Filed: 08/15/2008
• Published: 06/23/2011
• Est. Priority Date: 08/17/2007
• Status: Active Grant

• Current Assignee: Textron Innovations Incorporated (Textron Inc)
• Original Assignee: Textron Innovations Incorporated (Textron Inc)
• Inventors: Clanton, Carson Blaine, Lewis, George Steven
• Granted Patent: US 9,202,098 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: B64D 45/00 Aircraft indicators or protectors not otherwise provided for camouflage F41H3/00
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G01D 2207/10 Displays which are primarily used in aircraft or display aircraft-specific information
• CPC Class Codes: G01D 3/08 with provision for safeguarding the apparatus, e.g. against abnormal operation, against breakdown
• CPC Class Codes: G01D 7/02 Indicating value of two or more variables simultaneously
• CPC Class Codes: G01P 1/08 Arrangements of scales, pointers, lamps or acoustic indicators, e.g. in automobile speedometers
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 2201/02 Recognising information on displays, dials, clocks
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera

Abstract

A system for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls includes an image acquisition sensor operable to capture images of at least one of the instruments, annunciators, and controls; and a logic processing unit operable to decode the images captured by the image acquisition sensor and interpret the decoded images to determine a state of the at least one of the instruments, annunciators, and controls.

Claims

1. A system for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls, comprising:

an image acquisition sensor operable to capture images of at least one of the instruments, annunciators, and controls;

a logic processing unit operable to decode the images captured by the image acquisition sensor and interpret the decoded images to determine a state of the at least one of the instruments, annunciators, and controls.

2. The system of claim 1, wherein the logic processing unit is operable to store the state of the at least one of the instruments, annunciators, and controls.

3. The system of claim 1, wherein the logic processing unit is operable to store time-based states of the at least one of the instruments, annunciators, and controls.

4. The system of claim 1, wherein the logic processing unit creates a pseudo-parameter from the state of the at least one of the instruments, annunciators, and controls.

5. The system of claim 1, further comprising:

a ground data processor operably associated with the logic processing unit.

6. The system of claim 1, wherein the human readable instruments, annunciators, and controls are objects in an aircraft cockpit.

7. The system of claim 6, wherein the aircraft is a rotorcraft.

8. A method for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls, comprising:

reading a configuration file to provide a definition of a target region of an image to be interpreted of at least one of the instruments, annunciators, and controls to be interpreted;

opening an output file for storing the image to be interpreted;

determining whether a captured image to be interpreted exists;

if the captured image exists;

decoding the captured image;

determining a state of the at least one of the instruments, annunciators, and controls based upon the decoded captured image and the definition in the configuration file; and

writing the state of the at least one of the instruments, annunciators, and controls to the output file.

9. The method of claim 8, wherein the configuration file includes one or more of:

a number of the at least one of the instruments, annunciators, and controls to be interpreted;

types of the at least one of the instruments, annunciators, and controls to be interpreted;

locations of the at least one of the instruments, annunciators, and controls to be interpreted;

a definition of a scan region of the at least one of the instruments, annunciators, and controls to be interpreted;

value maps associated with the locations of the at least one of the instruments, annunciators, and controls to be interpreted;

a mathematical representation of a reference region of the image for use in image registration;

a mathematical representation of reference patterns used to determine states of the at least one of the instruments, annunciators, and controls to be interpreted;

chromacity and intensity information for use in interpreting illumination status for an annunciator;

chromacity and intensity information for use in interpreting background versus foreground information of the image; and

startup parameters and preferences.

10. The method of claim 8, wherein the output file includes a time history of parameters interpreted and analyzed based upon definitions read from the configuration file.

11. The method of claim 8, wherein decoding the image results in a two-dimensional pixel bitmap or raster stored as a two-dimensional array.

12. The method of claim 8, wherein determining the state of the at least one of the instruments, annunciators, and controls is accomplished by:

registering the captured image with the definition of the target region of the image to be interpreted;

adjusting coordinates of the target region defined by the configuration file based upon registering the captured image;

scanning each pixel of the registered, captured image;

applying a median filter to each pixel of the registered, captured image;

segmenting the target region into foreground and background; and

picking foreground pixels corresponding to a variable feature of the at least one of the instruments, annunciators, and controls to be interpreted.

13. The method of claim 12, wherein registering the captured image is accomplished by:

computing a Fourier transform of the captured image;

element-wise multiplying a reference Fourier transform conjugate from the configuration file with the Fourier transform of the captured image;

computing an inverse Fourier transform of the multiplied reference Fourier transform conjugate and the Fourier transform of the captured image;

finding row and column indices of a maximum value in an array resulting from the inverse Fourier transform; and

storing the row and column indices.

14. The method of claim 13, further comprising:

element-wise normalizing the multiplied reference Fourier transform conjugate and the Fourier transform of the captured image prior to computing the inverse Fourier transform.

15. The method of claim 12, wherein segmenting the target region into foreground and background is accomplished by:

computing a threshold value for the target region based upon results of applying the median filter to each pixel of the registered, captured image;

retrieving an intensity of each pixel in the target area of the captured image;

comparing the intensity of each pixel to the threshold value; and

classifying each pixel as foreground or background.

16. The method of claim 12, wherein picking foreground pixels corresponding to a variable feature of the at least one of the instruments, annunciators, and controls to be interpreted is accomplished by:

windowing the captured image around the target area;

registering the windowed area of the captured image with the definition of the target region of the image to be interpreted; and

determining the state of the at least one of the instruments, annunciators, and controls to be interpreted based on the result of registering the windowed area.

17. Software for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls, the software being embodied in computer-readable media and when executed operable to:

read a configuration file to provide a definition of a target region of an image to be interpreted of at least one of the instruments, annunciators, and controls to be interpreted;

open an output file for storing the image to be interpreted;

determine whether a captured image to be interpreted exists;

if the captured image exists;

decode the captured image;

determine a state of the at least one of the instruments, annunciators, and controls based upon the decoded captured image and the definition in the configuration file; and

write the state of the at least one of the instruments, annunciators, and controls to the output file.

18. The software of claim 17, wherein the configuration file includes one or more of:

a number of the at least one of the instruments, annunciators, and controls to be interpreted;

types of the at least one of the instruments, annunciators, and controls to be interpreted;

locations of the at least one of the instruments, annunciators, and controls to be interpreted;

a definition of a scan region of the at least one of the instruments, annunciators, and controls to be interpreted;

value maps associated with the locations of the at least one of the instruments, annunciators, and controls to be interpreted;

a mathematical representation of a reference region of the image for use in image registration;

a mathematical representation of reference patterns used to determine states of the at least one of the instruments, annunciators, and controls to be interpreted;

chromacity and intensity information for use in interpreting illumination status for an annunciator;

chromacity and intensity information for use in interpreting background versus foreground information of the image; and

startup parameters and preferences.

19. The software of claim 17, wherein the output file includes a time history of parameters interpreted and analyzed based upon definitions read from the configuration file.

20. The software of claim 17, wherein the software, when executed, is operable to decode the image results in a two-dimensional pixel bitmap or raster stored as a two-dimensional array.

21. The software of claim 17, wherein the software, when executed, determines the state of the at least one of the instruments, annunciators, and controls by:

registering the captured image with the definition of the target region of the image to be interpreted;

adjusting coordinates of the target region defined by the configuration file based upon registering the captured image;

scanning each pixel of the registered, captured image;

applying a median filter to each pixel of the registered, captured image;

segmenting the target region into foreground and background; and

picking foreground pixels corresponding to a variable feature of the at least one of the instruments, annunciators, and controls to be interpreted.

22. The software of claim 21, wherein the software, when executed, registers the captured image by:

computing a Fourier transform of the captured image;

element-wise multiplying a reference Fourier transform conjugate from the configuration file with the Fourier transform of the captured image;

computing an inverse Fourier transform of the multiplied reference Fourier transform conjugate and the Fourier transform of the captured image;

finding row and column indices of a maximum value in an array resulting from the inverse Fourier transform; and

storing the row and column indices.

23. The software of claim 22, wherein the software, when executed, registers the captured image by:

element-wise normalizing the multiplied reference Fourier transform conjugate and the Fourier transform of the captured image prior to computing the inverse Fourier transform.

24. The software of claim 21, wherein the software, when executed, segments the target region into foreground and background by:

computing a threshold value for the target region based upon results of applying the median filter to each pixel of the registered, captured image;

retrieving an intensity of each pixel in the target area of the captured image;

comparing the intensity of each pixel to the threshold value; and

classifying each pixel as foreground or background.

25. The method of claim 21, wherein the software, when executed, picks foreground pixels corresponding to a variable feature of the at least one of the instruments, annunciators, and controls to be interpreted is accomplished by:

windowing the captured image around the target area;

registering the windowed area of the captured image with the definition of the target region of the image to be interpreted; and

determining the state of the at least one of the instruments, annunciators, and controls to be interpreted based on the result of registering the windowed area.