Adaptive instrument and operator control recognition

• US 8,319,665 B2
• Filed: 02/20/2009
• Issued: 11/27/2012
• Est. Priority Date: 02/20/2009
• Status: Active Grant

• Current Assignee: Appareo Systems LLC (AGCO Corporation)
• Original Assignee: Appareo Systems LLC (AGCO Corporation)
• Inventors: Weinmann, Robert V., Allen, Robert M., Wiig, Johan A., Gelinske, Joshua N.
• Primary Examiner(s): TRIEU, VAN THANH
• Application Number:
  US12/390,146
• Publication Number:
  US 20100214130A1
• Time in Patent Office: 1,376 Days
• Field of Search: 340/945, 340/947, 340/955, 340/956, 340/963, 340/964, 340/971, 340/973, 340/974, 340/978, 340/982, 340/937, 340/438, 348/117, 348/143, 348/148, 382/209, 382/217, 725/108
• US Class Current: 340/937
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 2201/02 Recognising information on displays, dials, clocks

Abstract

A system and method of acquiring information from an image of an instrument panel of a vehicle in real time wherein at least one imaging device with advanced light metering capabilities is placed aboard a vehicle, a computer processor means is provided to control the imaging device and the advanced light metering capabilities, the advanced light metering capabilities are used to capture an image of at least a portion of the instrument panel, such as a gauge or operator control, and image recognition algorithms are used to identify the current state of the imaged portion of the instrument panel.

Claims

1. A method of acquiring information from an image within a vehicle comprising the steps of:

providing at least one imaging device aboard said vehicle, wherein said imaging device comprises individual raw pixel elements;

providing a computer processor to control said imaging device;

capturing image data representing an area within said vehicle using said raw pixel elements with said imaging device;

applying one or more advanced light metering techniques to at least a subset of said raw pixel elements to create adjusted raw pixel elements;

converting said adjusted raw pixel elements to an image;

inputting said image to said computer processor;

identifying with said computer processor a state of said image; and

said computer processor providing an output corresponding to said image state.

2. The method of claim 1 wherein all steps are performed in real time while said vehicle is in operation.

3. The method of claim 1 further comprising the steps of:

analyzing said image state with a rules engine executing on said computer processor;

determining if said image state indicates that said vehicle is in violation of a condition defined by said rules engine; and

initiating an appropriate response to said violation.

4. The method of claim 3 wherein said rules engine comprises aircraft flight profile rules as used by a flight operations quality assurance (FOQA) program.

5. The method of claim 3 wherein said step of initiating an appropriate response includes at least one of:

sounding an aural alarm;

displaying a visual alarm; and

reporting the condition to an off-board station by means of a telemetry device.

6. The method of claim 1, which includes the steps of:

providing said imaging device with advanced light metering capabilities chosen from among the group comprising spot metering, average metering and center-weighted average metering; and

controlling said light metering capabilities with said computer processor.

7. The method of claim 6 wherein said step of using advanced light metering capabilities to capture an image further includes applying said advanced light metering capabilities directly to raw pixel data before an image file is created.

8. The method of claim 1 wherein said image includes an object of interest comprising at least a portion of an instrument panel of said vehicle.

9. The method of claim 8 wherein said at least a portion of said instrument panel is a feature selected from the group comprising:

mechanical gauge, digital readout, status light, functional switch, computer display, and operator control.

10. The method of claim 1, which includes the steps of:

calibrating the imaging device with an adaptive imaging module;

acquiring a test image of an object of interest with said adaptive imaging module; and

identifying coordinates for said object of interest in said test image.

11. The method of claim 10, which includes the steps of:

providing an object library comprising information corresponding to images of pre-identified objects of interest;

determining if an object of interest is included in the object library;

if said object of interest is found in the object library, storing the object of interest configuration from said object library with said computer processor; and

if said object of interest is not found in the object library, identifying and storing its configuration and operational characteristics with said computer processor.

12. The method of claim 11 wherein said object library includes pre-identified objects of interest corresponding to specific types of vehicles and said method includes the step of selecting the vehicle type from said object library.

13. The method of claim 1, which includes the steps of:

providing raw image data of an object of interest from said imaging device;

applying a low-pass filter to said raw image data to remove image noise therefrom;

using an edge detection algorithm for identifying points in said image data at which the image brightness changes sharply or has detectable discontinuities;

applying a binary hard limiter to convert an edge-only image to a binary image of said object of interest; and

providing an output corresponding to a state of said object of interest binary image.

14. The method of claim 13, which includes the step of:

applying a high-pass filter to perform edge detection on said image data.

15. The method of claim 13, which includes the step of:

applying an image differentiator to perform edge detection on said image data.

16. The method of claim 13, which includes the steps of:

creating a fiducial image of an object of interest;

storing with said computer processor said fiducial image;

providing a binary image with said imaging device; and

aligning said binary image detected by said imaging device by comparing and cross-correlating said binary image with said fiducial image.

17. The method of claim 16, which includes the steps of:

applying a mask to said aligned binary image to isolate a portion of said object of interest;

said imaging device providing an input to said computer processor corresponding to said isolated portion of said object of interest; and

said computer processor analyzing a state of said isolated portion of said object of interest and providing a corresponding output.

18. The method of claim 17, which includes the steps of:

determining a state of the object of interest using either;

synthetic images of the isolated portion of the object of interest for comparison with the masked binary image;

or a linear regression to fit the points (pixels) from the masked binary image to determine the object of interest state.

19. The method of claim 1, which includes the steps of:

providing a configuration file for an object of interest with limits of travel of a moving or changing part of the object of interest; and

determining an operating condition of the vehicle by comparing the object of interest binary file with the configuration file.

20. The method of claim 1, which includes the steps of:

using a set-up utility to create a fiducial image comprising multiple individual fiducial images of multiple objects of interest; and

using said set-up utility to create a feature mask for each individual object of interest.

21. The method of claim 1, which includes the additional steps of:

receiving geospatial data corresponding to a geospatial position, velocity or attitude of said vehicle;

combining said geospatial data with data from said imaging device corresponding to a state of an object of interest in said vehicle to create a fused sensor value;

providing a rules engine corresponding to operating characteristics of said vehicle;

comparing with said computer processor said fused sensor value with said rules engine;

detecting an exceedance of said rules engine based on said comparison with said fused sensor value; and

providing an event response comprising at least one of;

recording said fused sensor value corresponding to said event;

recording a video from said imaging device output;

communicating the event and/or the fused sensor value offboard the vehicle via a telemetry device; and

communicating the event and/or the fused sensor value offboard the vehicle via a wide-area network.

22. The method of claim 21, which includes the steps of:

providing a global navigation satellite system (GNSS) receiver including a GNSS antenna mounted on said vehicle;

providing an inertial measurement unit (IMU) mounted on said vehicle;

connecting said GNSS receiver and said IMU to said computer processor;

providing input signals to said computer processor from said GNSS receiver and said IMU corresponding to geospatial positions and attitudes of said vehicle;

said computer processor calculating the location and orientation of said vehicle using said GNSS receiver and said IMU signals; and

said computer processor combining said vehicle location and orientation information with said extracted information to create said fused sensor value.

23. A method of acquiring information pertaining to the operation of a vehicle from images of objects of interest within the vehicle, including objects of interest on the instrument panel of the vehicle, which method comprises the steps of:

providing at least one imaging device aboard said vehicle;

providing a computer processor to control said imaging device;

calibrating the imaging device with an adaptive imaging module;

acquiring a test image of said object of interest with said adaptive imaging module;

identifying coordinates for said object of interest in said test image;

providing an object library comprising information corresponding to images of pre-identified objects corresponding to specific types of vehicles;

determining if an object of interest is included in the object library;

selecting the vehicle type from said object library;

if said object of interest is found in the object library, storing the object of interest configuration from said object library with said computer processor;

if said object of interest is not found in the object library, identifying and storing its configuration and operational characteristics with said computer processor;

providing raw image data of objects of interest from said imaging device;

applying a low-pass filter to said raw image data to remove image noise therefrom;

using an edge detection algorithm for identifying points in said image data at which the image brightness changes sharply or has detectable discontinuities;

applying a binary hard limiter to convert edge-only images to binary images of said objects of interest;

providing an output from said computer processor corresponding to a state of said objects of interest binary images;

applying either a high-pass filter or an image differentiator to perform edge detection on said image data;

using a set-up utility to create multiple individual fiducial images of multiple objects of interest;

storing with said computer processor said fiducial images;

providing binary images of said objects of interest with said imaging device;

aligning said binary images detected by said imaging device by comparing and cross-correlating said binary images with said fiducial images;

using said set-up utility to create a feature mask for each individual object of interest;

applying a mask to said aligned binary images to isolate portions of said objects of interest;

said imaging device providing an input to said computer processor corresponding to said isolated portions of said objects of interest;

said computer processor analyzing a state of said isolated portions of said objects of interest and providing a corresponding output;

determining states of the objects of interest using either;

synthetic images of the isolated portions of the objects of interest for comparison with the masked binary images;

or linear regressions to fit the points (pixels) from the masked binary images to determine the objects of interest states;

providing configuration files for the objects of interest with limits of travel of moving or changing parts of the objects of interest;

determining operating conditions of the vehicle by comparing the objects of interest binary files with the configuration files;

receiving geospatial data corresponding to a geospatial position, velocity or attitude of said vehicle;

combining said geospatial data with data from said imaging device corresponding to states of objects of interest in said vehicle to create fused sensor values;

providing a rules engine corresponding to operating characteristics of said vehicle;

comparing with said computer processor said fused sensor values with said rules engine;

detecting an exceedance(s) of said rules engine based on said comparisons with said fused sensor values; and

providing an event response(s) comprising at least one of;

recording said fused sensor value corresponding to said event;

recording a video from said imaging device output;

communicating the event and/or the fused sensor value offboard the vehicle via a telemetry device; and

communicating the event and/or the fused sensor value offboard the vehicle via a wide-area network.

24. A system for acquiring information from an image within a vehicle, which system comprises:

a software-controlled imaging device mounted within said vehicle;

said imaging device including individual raw pixel elements;

a computer processor connected to and adapted for controlling said imaging device;

said computer processor including a memory module;

said imaging device capturing an image of said object of interest and providing imaging data as an input to said computer processor;

said computer processor being adapted for storing said imaging data in said memory module;

said computer processor using said imaging data to determine an image state of said vehicle and providing an output corresponding thereto;

said imaging device includes advanced light metering capabilities chosen from among the group comprising spot metering, average metering and center-weighted average metering; and

said computer processor being adapted for controlling said light metering capabilities.

25. The system of claim 24 further comprising:

a rules engine executing on said computer processor and adapted for analyzing said image state; and

said computer processor being adapted for determining if said image state indicates that said vehicle is in violation of a condition defined by said rules engine and initiating an appropriate response to said violation.

26. The system of claim 25 wherein said rules engine comprises aircraft flight profile rules as used by a flight operations quality assurance (FOQA) program.

27. The method of claim 25 wherein said appropriate response includes at least one of:

sounding an aural alarm;

displaying a visual alarm; and

reporting the condition to an off-board station by means of a telemetry device.

28. The system of claim 24 wherein said imaging device is adapted for using advanced light metering capabilities to capture an image and for applying said advanced light metering capabilities directly to raw pixel data before an image file is created.

29. The system of claim 24, which includes:

an adaptive imaging module adapted for calibrating the imaging device;

said adaptive imaging module being adapted for acquiring a test image of an object of interest and identifying coordinates for said object of interest in said test image.

30. The system of claim 29, which includes:

an object library stored in said memory module and comprising information corresponding to images of pre-identified objects of interest;

means for determining if an object of interest is included in the object library;

means for storing the object of interest configuration from said object library with said computer processor if said object of interest is found in the object library; and

means for identifying and storing its configuration and operational characteristics with said computer processor if said object of interest is not found in the object library.

31. The system of claim 30 wherein said object library includes pre-identified objects of interest corresponding to specific types of vehicles and said computer processor is adapted for selecting the vehicle type from said object library.

32. The system of claim 24, which includes:

said imaging device being adapted for providing raw image data of an object of interest;

a low-pass filter connected to said imaging device and adapted for removing image noise from said role image data;

said computer processor including an edge detection algorithm for identifying points in said image data at which the image brightness changes sharply or has detectable discontinuities;

said computer processor including a binary hard limiter adapted for converting an edge-only image to a binary image of said object of interest; and

said computer processor adapted for providing an output corresponding to a state of said object of interest binary image.

33. The system of claim 32, which includes:

a high-pass filter connected to said imaging device and adapted for performing edge detection on said image data.

34. The system of claim 32, which includes:

an image differentiator connected to said imaging device and adapted for performing edge detection on said image data.

35. The system of claim 24, which includes:

said computer processor being adapted for;

creating and storing a fiducial image of an object of interest;

said imaging device being adapted for providing a binary image of said object of interest; and

said computer processor and adapted for aligning said binary image detected by said imaging device by comparing and cross-correlating said binary image with said fiducial image.

36. The system of claim 35, which includes:

said computer processor being adapted for applying a mask to said aligned binary image to isolate a portion of said object of interest;

said imaging device being adapted for providing an input to said computer processor corresponding to said isolated portion of said object of interest; and

said computer processor being adapted for analyzing a state of said isolated portion of said object of interest and providing a corresponding output.

37. The system of claim 36, which includes:

means for determining a state of the object of interest using either;

synthetic images of the isolated portion of the object of interest for comparison with the masked binary image;

or a linear regression to fit the points (pixels) from the masked binary image to determine the object of interest state.

38. The system of claim 24, which includes:

means for providing a configuration file for an object of interest with limits of travel of a moving or changing part of the object of interest; and

means for determining an operating condition of the vehicle by comparing the object of interest binary file with the configuration file.

39. The system of claim 24, which includes:

a set-up utility adapted for creating a fiducial image comprising multiple individual fiducial images of multiple objects of interest; and

said set-up utility being adapted for creating a feature mask for each individual object of interest.

40. The system of claim 24, which includes:

means for receiving geospatial data corresponding to a geospatial position, velocity or attitude of said vehicle;

means for combining said geospatial data with data from said imaging device corresponding to a state of an object of interest in said vehicle to create a fused sensor value;

means for providing a rules engine corresponding to operating characteristics of said vehicle;

means for comparing with said computer processor said fused sensor value with said rules engine;

means for detecting an exceedance of said rules engine based on said comparison with said fused sensor value; and

means for providing an event response comprising at least one of;

recording said fused sensor value corresponding to said event;

recording a video from said imaging device output;

communicating the event and/or the fused sensor value offboard the vehicle via a telemetry device; and

communicating the event and/or the fused sensor value offboard the vehicle via a wide-area network.

41. The system of claim 40, further comprising:

a global navigation satellite system (GNSS) receiver including a GNSS antenna, said receiver being mounted on or associated with said vehicle;

an inertial measurement unit (IMU) mounted on said vehicle;

said GNSS receiver and said IMU being connected to said computer processor and adapted for providing input signals thereto corresponding to geospatial positions and attitudes of said vehicle;

said computer processor being adapted for calculating the location and orientation of said vehicle using said GNSS receiver and said IMU signals; and

wherein said location and orientation are combined with said extracted information to create a fused sensor value.

42. The system of claim 41 further comprising a rules engine executing on said computer processing means, wherein said rules engine determines if said fused sensor value indicates that said vehicle is in violation of a condition defined by said rules engine.

43. The system of claim 42 further comprising a telemetry device for transmitting said extracted information and said violation from said vehicle.

44. The system of claim 24 wherein said image includes an object of interest comprising at least a portion of the instrument panel of said vehicle.

45. The system of claim 44 wherein said at least a portion of said instrument panel is a feature selected from the group comprising:

mechanical gauge, digital readout, status light, functional switch, and operator control.