Method and Apparatus for Providing Driver Information Via Audio and Video Metadata Extraction

• US 20180046870A1
• Filed: 08/09/2017
• Published: 02/15/2018
• Est. Priority Date: 08/10/2016
• Status: Active Grant

• Current Assignee: Xevo, Inc. (Lear Corporation)
• Original Assignee: Surround.io Corporation (Lear Corporation)
• Inventors: Cordell, John Palmer, Welland, Robert Victor, McKelvie, Samuel James, Ludwig, John Hayes
• Granted Patent: US 10,540,557 B2
• Time in Patent Office: Days
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 1/04 mounted inside vehicle B60R1/025, B60R1/08 take precedence
• CPC Class Codes: B60R 1/062 with remote control for adjusting position B60R1/0607 takes precedence
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/8006 for monitoring and displaying scenes of vehicle interior, e.g. for monitoring passengers or cargo
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3605 Destination input or retrieval
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G05D 1/0214 in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 2201/10 : Recognition assisted with metadata
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 : Sensors therefor
• CPC Class Codes: G08B 21/06 : indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0116 : from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/04 : using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/096811 : where the route is computed offboard
• CPC Class Codes: G08G 1/096861 : where the immediate route instructions are output to the driver, e.g. arrow signs for next turn
• CPC Class Codes: G08G 1/096888 : where input information is obtained using learning systems, e.g. history databases
• CPC Class Codes: G08G 1/0969 : having a display in the form of a map
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/44 : for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

A method and/or system is able to provide driver fingerprint via metadata extraction managed by a driver rating (“DR”) model trained by a machine learning center (“MLC”) coupled to a cloud based network (“CBN”). In one embodiment, a DR system includes a set of outward facing cameras, a set of inward facing cameras, and a vehicle onboard computer (“VOC”). The set of outward facing cameras mounted on a vehicle is used to collect external images representing a surrounding environment in which the vehicle operates. The set of inward facing cameras mounted in the vehicle is used to collect internal images including operator body expression representing at least operator'"'"'s attention. The VOC is configured to determine the identity of operator and current operating style in response to the collected internal images, the collected external images, and historical stored data.

Claims

1. A system for providing driver fingerprint via metadata extraction managed by a driver rating (“

DR”

) model trained by a machine learning center (“

MLC”

) coupled to a cloud based network (“

CBN”

), comprising;

a plurality of outward facing cameras mounted on a vehicle collecting external images representing a surrounding environment in which the vehicle operates;

a plurality of inward facing cameras mounted in the vehicle collecting internal images including operator body expression representing at least operator'"'"'s attention; and

a vehicle onboard computer (“

VOC”

), coupled to CBN, the plurality of outward facing cameras, and the plurality of inward facing cameras, configured to determine identity of operator and current operating style in response to the collected internal images, the collected external images, and historical stored data.

2. The system of claim 1, wherein the CBN is wirelessly coupled to the VOC and configured to correlate and generate labeled data associated with driver fingerprint based on historical DR cloud data, the collected internal images, and the collected external images.

3. The system of claim 1, wherein the MLC is coupled to the VOC and configured to train and improve the DR model based on the labeled data from the cloud based network.

4. The system of claim 1, wherein the plurality of outward facing cameras is configured to capture real-time images as the vehicle moves across a geographical area.

5. The system of claim 1, wherein the plurality of inward facing cameras is configured to extract metadata associated with operator head pose, gaze direction, and looking at a mobile device.

6. The system of claim 1, wherein the DR model generates a rating model for the operator based on calculated feature vectors via the collected internal images, the collected external images, and historical stored data.

7. The system of claim 6, wherein the vectors indicates one of aggressive driving style, inattentive driving style, distracted driving style, and driving under influence.

8. The system of claim 1, further comprising a plurality of audio sensors coupled to the VOC and configured to provide metadata relating to audio data.

9. The system of claim 8, wherein the plurality of audio sensors includes exterior audio sensors collecting exterior sound outside of the vehicle and interior audio sensors collecting interior sound inside of the vehicle.

10. The system of claim 1, wherein the collecting external images includes obtaining real-time images relating to at least one of road, buildings, traffic lights, pedestrian, and retailers.

11. The system of claim 1, wherein the collecting internal images includes a set of interior sensors obtaining data relating to at least one of operator'"'"'s eyes, facial expression, driver, and passage.

12. The system of claim 11, wherein the obtaining data relating to at least one of operator'"'"'s eyes, facial expression, driver, and passage includes at least one camera is capable of detecting direction of where the operator is looking.

13. The system of claim 1, wherein the VOC is configured to communicate with a virtuous cycle to modify the DR model.

14. A method for providing driver rating via metadata extraction utilizing a virtuous cycle including sensors, machine learning center, and cloud based network, comprising:

activating a set of outward facing cameras mounted on a vehicle for recording external surrounding images representing a geographic environment in which the vehicle operates;

enabling at least a portion of inward facing cameras mounted in the vehicle for collecting interior images of the vehicle;

calculating a driver style vector in accordance with external surrounding images, the internal images, and historical data; and

generating a driver rating associated to operator by performing a driver style analysis conducted by a driver rating (“

DR”

) model trained by the virtuous cycle.

15. The method of claim 14, further comprising tracking surrounding environmental event in accordance with the external surrounding images and historical data supplied by the virtuous cycle.

16. The method of claim 14, further comprising generating a process of adaptive advice and assistance by an adaptive driver assistant (“

ADA”

) model in response to external surrounding images, the internal images, and historical data.

17. The method of claim 14, further comprising providing the driver rating to a subscriber requesting driver fingerprint.

18. A method configured to provide adaptive driver assistant (“

ADA”

) utilizing a virtuous cycle containing sensors, machine learning center (“

MLC”

), and cloud based network (“

CBN”

), comprising;

monitoring operator body language of an operator captured by a set of interior sensors and capturing surrounding information observed by a set of exterior sensors onboard a vehicle as the vehicle is in motion;

selectively recording data relating to the operator body language and the surrounding information in accordance with a containerized ADA model generated by MLC;

detecting an abnormal driving activity in accordance with the operator body language and surrounding information by the ADA model while the vehicle is in driving mode; and

generating a process of adaptive advice and assistance by ADA model in response to the surrounding images, the operator body language, and historical data.

19. The method of claim 18, further comprising rewinding recorded operator body language and the surrounding information leading up to detection of the abnormal driving activity and generating labeled data associated with the abnormal driving activity.

20. The method of claim 19, further comprising uploading the labeled data to the cloud based network for facilitating ADA model training at the MLC via a virtuous cycle.