Method and apparatus for providing driver information via audio and video metadata extraction

• US 10,540,557 B2
• Filed: 08/09/2017
• Issued: 01/21/2020
• Est. Priority Date: 08/10/2016
• Status: Active Grant

• Current Assignee: Xevo, Inc. (Lear Corporation)
• Original Assignee: Xevo, Inc. (Lear Corporation)
• Inventors: Cordell, John Palmer, Welland, Robert Victor, McKelvie, Samuel James, Ludwig, John Hayes
• Primary Examiner(s): Moyer, Andrew M
• Application Number:
  US15/672,832
• Publication Number:
  US 20180046870A1
• Time in Patent Office: 895 Days
• Field of Search: 382104
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 1/04 mounted inside vehicle B60R1/025, B60R1/08 take precedence
• CPC Class Codes: B60R 1/062 with remote control for adjusting position B60R1/0607 takes precedence
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/8006 for monitoring and displaying scenes of vehicle interior, e.g. for monitoring passengers or cargo
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3605 Destination input or retrieval
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G05D 1/0214 in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 2201/10 : Recognition assisted with metadata
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 : Sensors therefor
• CPC Class Codes: G08B 21/06 : indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0116 : from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/04 : using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/096811 : where the route is computed offboard
• CPC Class Codes: G08G 1/096861 : where the immediate route instructions are output to the driver, e.g. arrow signs for next turn
• CPC Class Codes: G08G 1/096888 : where input information is obtained using learning systems, e.g. history databases
• CPC Class Codes: G08G 1/0969 : having a display in the form of a map
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/44 : for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

A method and/or system is able to provide driver fingerprint via metadata extraction managed by a driver rating (“DR”) model trained by a machine learning center (“MLC”) coupled to a cloud based network (“CBN”). In one embodiment, a DR system includes a set of outward facing cameras, a set of inward facing cameras, and a vehicle onboard computer (“VOC”). The set of outward facing cameras mounted on a vehicle is used to collect external images representing a surrounding environment in which the vehicle operates. The set of inward facing cameras mounted in the vehicle is used to collect internal images including operator body expression representing at least operator'"'"'s attention. The VOC is configured to determine the identity of operator and current operating style in response to the collected internal images, the collected external images, and historical stored data.

Claims

1. A system for providing a current operating style of a driver of a vehicle, comprising:

a plurality of outward facing cameras mounted on the vehicle collecting external images representing a surrounding environment in which the vehicle operates;

a plurality of inward facing cameras mounted in the vehicle collecting internal images including operator body expression representing at least an attention of the driver; and

a vehicle onboard computer (“

VOC”

), coupled to the plurality of outward facing cameras and the plurality of inward facing cameras, configured to;

determine a plurality of current operator characteristics of the driver based on an analysis of the external images and the internal images;

calculate a feature vector for each of the plurality of current operator characteristics; and

determine the current operating style of the driver by plotting the feature vector of each of the plurality of current operator characteristics onto a multi-dimensional space, wherein different areas in the multi-dimensional space are associated with different operating styles.

2. The system of claim 1, wherein the VOC is further configured to:

determine a second plurality of current operator characteristics of the driver based on an analysis of the external images and the internal images,

calculate a second feature vector for each of the second plurality of current operator characteristics;

determine a second current operating style of the driver by plotting the second feature vector of each of the second plurality of current operator characteristics onto the multi-dimensional space; and

determine a change in driving style of the driver based on a delta distance between the current operating style and the second current operating style in the multi-dimensional space.

3. The system of claim 1, wherein the VOC determines the change in driving style by being configured to:

determine a direction of movement from the current operating style to the second current operating style in the multi-dimensional space, wherein the direction of movement is associated with a particular change in driving style.

4. The system of claim 1, wherein the plurality of outward facing cameras is configured to capture real-time images as the vehicle moves across a geographical area.

5. The system of claim 1, wherein the plurality of inward facing cameras is configured to extract metadata associated with at least one of operator head pose, gaze direction, and looking at a mobile device.

6. The system of claim 1, wherein the different operating styles include aggressive driving style, inattentive driving style, distracted driving style, and driving under influence.

7. The system of claim 1, further comprising a plurality of audio sensors configured to provide metadata relating to audio data to the VOC, wherein the VOC further determines the plurality of current operator characteristics based on the audio data.

8. The system of claim 7, wherein the plurality of audio sensors includes exterior audio sensors collecting exterior sound outside of the vehicle and interior audio sensors collecting interior sound inside of the vehicle.

9. The system of claim 1, wherein the collecting external images includes obtaining real-time images relating to at least one of road, buildings, traffic lights, pedestrian, and retailers.

10. The system of claim 1, wherein the collecting internal images includes a set of interior sensors obtaining data relating to at least one of the driver'"'"'s eyes, the driver'"'"'s facial expression, a passenger'"'"'s vocal expressions, and a passenger'"'"'s movement.

11. The system of claim 10, wherein the plurality of inward facing cameras includes at least one camera capable of detecting direction of where the driver is looking.

12. A method for providing driver rating of a driver of a vehicle, comprising:

capturing, via one or more outward facing cameras mounted on the vehicle, external surrounding images representing a geographic environment in which the vehicle operates;

capturing, via one or more inward facing cameras mounted in the vehicle, interior images of the vehicle;

determining a plurality of current operator characteristics of the driver based on an analysis of the external surrounding images and the internal images;

calculating a feature vector for each of the plurality of current operator characteristics;

determining a current operating style of the driver by plotting the feature vector of each of the plurality of current operator characteristics onto a multi-dimensional space, wherein different areas in the multi-dimensional space are associated with different operating styles; and

generating a driver rating for the driver based on the current operating style and a driver rating model trained front historical driving data of the driver.

13. The method of claim 12, further comprising tracking surrounding environmental event in accordance with the external surrounding images and the historical driving data.

14. The method of claim 12, further comprising generating adaptive advice and assistance for the driver in response to the current operating style.

15. The method of claim 12, further comprising providing the driver rating to a subscriber requesting driver style information.