Method and system for driver monitoring by fusing contextual data with event data to determine context as cause of event

• US 10,065,652 B2
• Filed: 03/26/2016
• Issued: 09/04/2018
• Est. Priority Date: 03/26/2015
• Status: Active Grant

• Current Assignee: Lightmetrics Technologies Pvt. Ltd.
• Original Assignee: Lightmetrics Technologies Pvt. Ltd.
• Inventors: Shenoy, Ravi, A G, Krishna, Putraya, Gururaj, Ukil, Soumik, Uliyar, Mithun, Patwardhan, Pushkar
• Primary Examiner(s): Tweel, Jr., John A
• Application Number:
  US15/526,839
• Publication Number:
  US 20180001899A1
• Time in Patent Office: 892 Days
• Field of Search: 340439
• CPC Class Codes: B60W 2420/905 the sensor being an xyz axis sensor
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0808 Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0816 Indicating performance data, e.g. occurrence of a malfunction

Abstract

A method and system for driver monitoring by fusing contextual data with event data to determine context as cause of event is provided. The method includes detecting an event from event data received from one or more inertial sensors associated with a vehicle of a driver or with the driver. The method also includes determining a context from an audio or video feed received from one or more devices associated with the vehicle or the driver. Further, the method includes fusing the event with the context to determine the context as a cause of the event. In addition, the method includes assigning a score to a driver performance metric based at least on the context and the event.

Claims

1. A method for driver monitoring, the method comprising:

detecting an event from event data received from one or more inertial sensors associated with a vehicle of a driver or with the driver, wherein detecting the event comprises calculating an inertial sensor score si[n];

determining a context automatically in real time from an audio or video feed only received from one or more devices associated with the vehicle or the driver, wherein the context is determined locally in the vehicle without using a server remotely located to the vehicle and determining the context comprises calculating a score s[n] from the context;

fusing the event with the context to determine the context as a cause of the event automatically in real time and locally in the vehicle without using the server remotely located to the vehicle; and

assigning a score S[n] to a driver performance metric based at least on the context and the event, wherein S[n]=F(s[n], si[n]) with F( ) being a function of at least one of multiplication, addition, subtraction, or division, and S[n] being impacted positively or negatively based on the context and the event.

2. The method as claimed in claim 1, wherein the event comprises harsh braking and the context determined using the audio or video feed comprises at least one of:

the driver of the vehicle was driving properly but a driver of a nearby vehicle applied sudden brake or a vehicle in the vicinity suddenly stopped or moved in an unexpected way;

object coming in unforeseen circumstances in front, wherein the object can be visually recognized and is not meant to be on the road.

3. The method as claimed in claim 1, wherein the event comprises hard acceleration and the context determined using the audio or video feed comprises at least one of:

the driver accelerating to avoid an accident due to an unforeseen object or circumstances, wherein the object can be visually recognized and is not meant to be on the road; and

the driver overtaking.

4. The method as claimed in claim 1, wherein the event comprises hard acceleration and the context determined using the audio or video feed comprises:

another vehicle or driver moving in an unexpected manner, causing the driver to take protective or evasive action.

5. The method as claimed in claim 1, wherein the event comprises lane change and the context determined using the audio or video feed comprises at least one of:

the driver heard a vehicle honking; and

the driver swerving or cornering to avoid an accident due to an unforeseen object or circumstances, wherein the object can be visually recognized and is not meant to be on the road.

6. The method as claimed in claim 1, wherein the event comprises distraction of the driver and the context comprises at least one of:

chat with a passenger;

drowsiness of the driver;

driver showing head movement greater than a threshold;

usage of a mobile phone by the driver; and

any unexpected or unforeseen sounds coming from within the vehicle or outside the vehicle.

7. The method as claimed in claim 1, wherein the event comprises detection of non-compliance in driving and the context comprises distraction of the driver.

8. The method as claimed in claim 7, wherein the one or more inertial sensors include at least one head mounting device configured to be worn by the driver.

9. The method as claimed in claim 1, wherein

si[n]=max(0, 1−

|a|/const*g), where a=magnitude of acceleration, g=acceleration due to gravity, and const=constant term; and

s[n]=f(p, m), where n=time instant, p=prescribed value, m=measured value and f(p, m) is a function of p and m.

10. The method as claimed in claim 1 and further comprising:

inferring driving environment from the audio or video feed or the event data; and

updating maps data based at least on the inferring.

11. The method as claimed in claim 1 and further comprising:

estimating a driver profile using the event data and the audio or video feed; and

alerting the driver based at least on the estimation.

12. The method as claimed in claim 11, wherein estimating the driver profile comprises:

dividing the event data or the audio or video feed into T time sub-units and each sub-unit being represented by d dimensional feature vectors that characterize driving behaviour within that sub-unit;

concatenating the feature vectors from the T time sub-units to form an N-dimensional driver profile obtained by using a statistical or a neural network based modelling technique; and

wherein alerting comprises, alerting the driver when there is a deviation from the driver profile during a drive, the deviation being detected based on evaluating a feature vector obtained during the drive against the driver profile and estimating the likelihood or the probability that the feature vector belongs to the driver profile.

13. The method as claimed in claim 9, wherein assigning the score comprises:

assigning the score positively or negatively.

14. The method as claimed in claim 13, wherein assigning the score comprises:

checking a predetermined rule based at least on a machine learning system that decides whether the score is impacted positively or negatively given the context and the event, wherein extent of the impact caused by negative scoring is dependent on degree of non-compliance.

15. A system for driver monitoring, the system comprising:

an event detector for detecting, automatically in real time and locally in a vehicle without using a server remotely located to the vehicle, an event from event data received from one or more inertial sensors associated with the vehicle of a driver or with the driver, wherein the one or more inertial sensors comprise at least one head mounting device configured to be worn by the driver, wherein detecting the event comprises calculating an inertial sensor score si[n];

a context detector for determining, automatically in real time and locally in the vehicle without using the server remotely located to the vehicle, a context from an audio or video feed only received from one or more devices associated with the vehicle or the driver, wherein determining the context comprises calculating a score s[n] from the context;

an analyser for fusing, automatically in real time and locally in the vehicle without using the server remotely located to the vehicle, the event with the context to determine the context as a cause of the event; and

a scorer for assigning, automatically in real time and locally in the vehicle without using the server remotely located to the vehicle, a score S[n] to a driver performance metric based at least on the context and the event, wherein S[n]=F(s[n], si[n]) with F( ) being a function of at least one of multiplication, addition, subtraction, or division, and S[n] being impacted positively or negatively based on the context and the event.

16. A method for driver monitoring, the method comprising:

detecting a vehicle event, automatically in real time and locally in a vehicle without using a server remotely located to the vehicle, from vehicle event data, wherein the vehicle event indicates a change in state of the vehicle;

detecting a driver event, automatically in real time and locally in the vehicle without using the server remotely located to the vehicle, from driver event data, wherein the driver event indicates a change in state of a driver and is determined from an audio or video feed only;

detecting a compliance event, automatically in real time and locally in the vehicle without using the server remotely located to the vehicle, from compliance event data, wherein the compliance event indicates an event related to compliance;

analyzing automatically in real time and locally in the vehicle without using the server remotely located to the vehicle, the vehicle event data, the driver event data, and the compliance event data to determine relationship between the vehicle event data, the driver event data, and the compliance event data; and

assigning automatically in real time and locally in the vehicle without using the server remotely located to the vehicle, a score to a driver profile based at least on the determined relationship.

17. The method as claimed in claim 16, where in the vehicle event data, the driver event data, and the compliance event data are detected using one or more inertial sensors attached to the vehicle or the driver, wherein the one or more inertial sensors comprise at least one head mounting device configured to be worn by the driver.

18. The method as claimed in claim 16 and further comprising:

estimating the driver profile using at least two of the vehicle event data, the driver event data and the compliance event data; and

alerting the driver based at least on estimating.

19. The method as claimed in claim 16 and further comprising:

inferring driving environment from at least two of the vehicle event data, the driver event data and the compliance event data; and

updating maps data based at least on the inferring.

20. The method as claimed in claim 16, wherein each event data is obtained from one or more sensors or devices attached to the vehicle or the driver.