Artificial intelligence based vehicle dashboard analysis

• US 10,152,641 B2
• Filed: 01/20/2017
• Issued: 12/11/2018
• Est. Priority Date: 01/20/2017
• Status: Active Grant

• Current Assignee: Advanced Lab For Automotive Ventures, LLC
• Original Assignee: Jack Cooper Logistics, LLC (Jack Cooper Enterprises, Inc.)
• Inventors: Amico, Andrea, Prabhushankar, Mohit
• Primary Examiner(s): Le, Peter D
• Application Number:
  US15/411,543
• Publication Number:
  US 20180211122A1
• Time in Patent Office: 690 Days
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/31 Virtual images
• CPC Class Codes: B60K 2360/56 using mobile devices
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60K 37/00 Dashboards as road-vehicle superstructure sub-units B62D25/14
• CPC Class Codes: B60K 37/20 Dashboard panels
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 2201/02 Recognising information on displays, dials, clocks
• CPC Class Codes: G07C 2205/02 using a vehicle scan tool
• CPC Class Codes: G07C 5/004 Indicating the operating range of the engine economical driving B60R16/0236, F02B77/084; measuring liquid fuel of an engine G01F9/00
• CPC Class Codes: G07C 5/0825 using optical means
• CPC Class Codes: Y02T 10/40 Engine management systems

Abstract

A portable computing device equipped with an image capture device captures an image of a vehicle dashboard of a vehicle. Then, the portable computing device identifies the location of one or more components of the vehicle dashboard in the captured image. Based on the location of the one or more components, the portable computing device segments the captured image to obtain an image of each of the one or more components. Further, the portable computing device processes the images of the one or more components using one or more machine learning models to determine a reading associated with each of the one or more components. An accuracy of the readings is verified and responsively, the portable computing device inputs the readings in respective data fields of an electronic form. The readings associated with the one or more components of the vehicle dashboard represent data associated with the vehicle.

Claims

1. A system comprising:

a portable computing device comprising one or more image capture devices, wherein the portable computing device is configured to;

generate one or more machine learning models that are configured to obtain a reading associated with at least one component of a vehicle dashboard of a vehicle from an image of the vehicle dashboard;

capture, using the one or more image capture devices, the image of the vehicle dashboard;

determine an approximate location of the at least one component of the vehicle dashboard in the captured image;

segment the captured image to obtain an image of the at least one component of the vehicle dashboard based on the approximate location of the at least one component in the captured image;

responsively, process, using the one or more machine learning models, the image of the at least one component to obtain the reading associated with the at least one component, the reading representative of data associated with the vehicle;

verify an accuracy of the reading associated with the at least one component that is obtained using the one or more machine learning models; and

responsive to a positive verification result, input the reading associated with the at least one component in a respective data field of an electronic form,

wherein responsive to a negative verification result, the portable computing device is configured to record the captured image as an exception and provide the captured image as feedback to the one or more machine learning models.

2. The system of claim 1, wherein the at least one component comprises one of a digital odometer, a warning light, an analog gauge, and a digital gauge.

3. The system of claim 1, wherein when the at least one component is a digital odometer, to process the image of the at least one component, the portable computing device is configured to:

determine if a unit of the reading associated with the digital odometer can be resolved from the captured image of the vehicle dashboard;

responsive to a negative determination, determine a location associated with the vehicle dashboard from a location metadata that is appended to the captured image of vehicle dashboard;

determine the unit associated with the reading of the digital odometer based on the location associated with the vehicle dashboard; and

responsive to a positive determination, input an image of the digital odometer that is segmented from the captured image of the vehicle dashboard to an odometer model to obtain the reading associated with the digital odometer, wherein the odometer model is one of the one or more machine learning models, and wherein the odometer model is configured to obtain the reading of the digital odometer from the image of the digital odometer.

4. The system of claim 1, wherein when the at least one component is an analog gauge, to process the image of the at least one component, the portable computing device is configured to:

input an image of the analog gauge that is segmented from the captured image of the vehicle dashboard to an analog gauge model,

wherein the analog gauge model is one of the one or more machine learning models, and wherein the analog gauge model is configured to determine a current position of an indicator of the analog gauge from the image of the analog gauge, wherein the indicator represents the reading of the analog gauge; and

determine the reading of the analog gauge based on the current position of the indicator of the analog gauge and a position of a maximum or a minimum value of the analog gauge.

5. The system of claim 1, wherein when the at least one component is a digital gauge, to process the image of the at least one component to obtain the reading associated with the at least one component, the portable computing device is configured to:

input an image of the digital gauge that is segmented from the captured image of the vehicle dashboard to a digital gauge model,

wherein the digital gauge model is one of the one or more machine learning models, and wherein the digital gauge model is configured to determine a current position of an indicator of the digital gauge from the image of the digital gauge, wherein the indicator represents the reading of the digital gauge; and

determine the reading of the digital gauge based on the current position of the indicator of the digital gauge and a position of a maximum or a minimum value of the digital gauge.

6. The system of claim 1, wherein when the at least one component is a warning light, to process the image of the at least one component to obtain the reading associated with the at least one component, the portable computing device is configured to:

input an image of the warning light that is segmented from the captured image of the vehicle dashboard to a warning light model,

wherein the warning light model is one of the one or more machine learning models, and

wherein the warning light model is configured to determine that the warning light is switched on and a condition of the vehicle represented by the warning light; and

responsive to a positive determination that the warning light is switched on in the image of warning light;

determine the condition of the vehicle represented by the warning light; and

generate and present an alert indicating the status of the warning light and the condition of the vehicle represented by the warning light.

7. The system of claim 1, wherein the approximate location of the at least one component of the vehicle dashboard in the captured image is determined based on a make and a model of the vehicle.

8. The system of claim 1, wherein the approximate location of the at least one component of the vehicle dashboard in the captured image is determined based on a user input.

9. The system of claim 7, wherein the make and the model of the vehicle is determined based on one of a user input and a vehicle identification number of the vehicle associated with the vehicle dashboard.

10. The system of claim 7, wherein the make and the model of the vehicle is determined using a vehicle type model of the one or more machine learning models that is configured to determine the make and the model of the vehicle based on the captured image of the vehicle dashboard.

11. A system comprising:

a portable computing device comprising one or more image capture devices, wherein the portable computing device is configured to;

capture, using the one or more image capture devices, an image of a vehicle dashboard of a vehicle;

determine an approximate location of an odometer, a digital gauge, an analog gauge, and a warning light of the vehicle dashboard in the captured image;

segment the captured image to obtain an image of the each of the odometer, the digital gauge, the analog gauge, and the warning light based on the approximate location of the odometer, the digital gauge, the analog gauge, and the warning light in the captured image;

responsively,

process, using a first machine learning model, the image of the odometer to determine a reading associated with the odometer,

process, using a second machine learning model, the image of the analog gauge to determine a reading associated with the analog gauge,

process, using a third machine learning model, the image of the digital gauge to determine a reading associated with the digital gauge, and

process, using a fourth machine learning model, the image of the warning light to determine a condition of the vehicle represented by the warning light;

verify an accuracy of the reading associated with the odometer, the reading associated with the digital gauge, the reading associated with the analog gauge, and the condition of the vehicle represented by the warning light; and

responsive to a positive verification result, input the readings in respective data fields of an electronic form,

wherein responsive to a negative verification result, record the captured image of the vehicle dashboard as an exception and provide the captured image as feedback to the first, second, third, and/or fourth machine learning model.

12. The system of claim 11:

wherein the first machine learning model is an odometer model that is trained to identify a reading associated with a digital odometer from an image of the digital odometer; and

wherein to process, using the first machine learning model, the image of the odometer, the portable computing device is configured to;

determine that the odometer is the digital odometer based on a make and a model of the vehicle;

responsive to a positive determination, input the image of the odometer to the odometer model;

determine, using the odometer model, each digit representing the reading associated with the odometer; and

output the reading of the odometer; and

determine a unit of the reading associated with the odometer based on one of the image of the odometer and location of the vehicle obtained from a location metadata appended to the captured image of the vehicle dashboard.

13. The system of claim 11:

wherein the second machine learning model is an analog gauge model that is trained to identify the reading associated with the analog gauge from the image of the analog gauge; and

wherein to process, using the second machine learning model, the image of the analog gauge, the portable computing device is configured to;

input the image of the analog gauge to the analog gauge model;

determine, using the analog gauge model, a position of an indicator of the analog gauge in the image of the analog gauge; and

determine the reading of the analog gauge based on the position of the indicator of the analog gauge and a position of a maximum or minimum value of the analog gauge.

14. The system of claim 11:

wherein the third machine learning model is a digital gauge model that is trained to identify the reading associated with the digital gauge from the image of the digital gauge; and

wherein to process, using the third machine learning model, the image of the digital gauge, the portable computing device is configured to;

input the image of the digital gauge to the digital gauge model;

determine, using the digital gauge model, a position of an indicator of the digital gauge in the image of the digital gauge; and

determine the reading of the digital gauge based on the position of the indicator of the digital gauge and a position of a maximum or minimum value of the digital gauge.

15. The system of claim 11:

wherein the fourth machine learning model is a warning light model that is trained to identify that the warning light is switched on and to identify the condition of the vehicle represented by the warning light from the image of the warning light; and

wherein to process, using the fourth machine learning model, the image of the warning light, the portable computing device is configured to;

input the image of the warning light to the warning light model;

determine, using the warning light model, that the warning light in the image of the warning light is switched on; and

responsively, determine the condition of the vehicle represented by the warning light, and generate an alert message indicating the status of the warning light.

16. A system comprising:

a portable computing device comprising at least one or more image capture devices and a wireless transceiver, wherein the portable computing device is configured to;

capture, using the one or more image capture devices, an image of a vehicle dashboard of a vehicle;

determine an approximate location of at least one component of the vehicle dashboard in the captured image;

segment the captured image to obtain an image of the at least one component of the vehicle dashboard based on the approximate location of the at least one component in the captured image; and

transmit, by the wireless transceiver, the image of the at least one component to a server;

the server that is communicably coupled to the portable computing device and configured to;

generate one or more machine learning models that are configured to obtain a reading associated with the at least one component of the vehicle dashboard from the image of the vehicle dashboard;

responsive to receiving the image of the at least one component, process, using the one or more machine learning models, the image of the at least one component to obtain the reading associated with the at least one component, the reading representative of data associated with the vehicle; and

transmit the reading associated with the at least one component to the portable computing device to verify an accuracy of the reading associated with the at least one component and input the reading associated with the at least one component in a respective data field of an electronic form responsive to a positive verification result.

17. The system of claim 16, wherein the at least one component comprises one of a digital odometer, a warning light, an analog gauge, and a digital gauge.