Driver behavior monitoring

• US 10,460,600 B2
• Filed: 02/21/2017
• Issued: 10/29/2019
• Est. Priority Date: 01/11/2016
• Status: Active Grant

• Current Assignee: Netradyne Inc.
• Original Assignee: Netradyne Inc.
• Inventors: Julian, David Jonathan, Agrawal, Avneesh
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Abimbola, Kehinde
• Application Number:
  US15/437,646
• Publication Number:
  US 20170200061A1
• Time in Patent Office: 980 Days
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G07C 5/0808 Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 for classifying traffic situation
• CPC Class Codes: G08G 1/0137 for specific applications
• CPC Class Codes: G08G 1/015 with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/052 with provision for determining speed or overspeed speed measuring in general G01P

Abstract

Systems and methods provide, implement, and use using a computer-vision based methods of context-sensitive monitoring and characterization of driver behavior. Additional systems and methods are provided for unsupervised learning of action values, monitoring of a driver'"'"'s environment, and transmitting visual information from a client to a server.

Claims

1. A method comprising:

initially receiving, by at least one processor of a computing device, first visual data captured by at least one camera associated with a monitoring vehicle;

identifying, by the at least one processor, a monitored vehicle from the first visual data;

receiving, by the at least one processor, second visual data captured by the at least one camera, wherein the second visual data is captured by the at least one camera subsequent to the capture of the first visual data;

identifying, by the at least one processor, the monitored vehicle from the second visual data based at least in part on a spatial location of the monitored vehicle within the first visual data and the second visual data;

determining, by the at least one processor, at least one change to an aspect of the monitored vehicle based on a comparison of the first visual data and the second visual data;

determining, by the at least one processor, an occurrence of an unsafe driving event comprising an unsafe following distance between the monitoring vehicle and the monitored vehicle, wherein the monitoring vehicle is following the monitored vehicle; and

determining, by the at least one processor, whether the monitoring vehicle or the monitored vehicle caused the occurrence of the unsafe driving event based at least in part on said at least one change.

2. The method of claim 1, further comprising identifying, by the at least one processor, at least one traffic control device from at least one of the first visual data or the second visual data.

3. The method of claim 2, wherein the at least one traffic control device is one or more lane marker.

4. The method of claim 3, further comprising:

determining, by the at least one processor, based at least in part on a spatial location of the one or more lane marker in the first visual data, that the monitored vehicle was in an adjacent lane with respect the monitoring vehicle at a first time; and

determining, by the at least one processor, based at least in part on a spatial location of the one or more lane marker in the second visual data, that the monitored vehicle was in a same lane with respect the monitoring vehicle at a second time.

5. The method of claim 4, wherein the said at least one change is a change of the monitored vehicle from the adjacent lane in the first visual data to the same lane in the second visual data.

6. The method of claim 5, wherein the monitored vehicle is determined to have caused the occurrence of the unsafe driving event.

7. The method of claim 3, further comprising determining, by the at least one processor, based at least in part on a spatial location of the one or more lane marker in the first visual data and the second visual data, that the monitoring vehicle and the monitored vehicle are in a same lane at a first time and a second time.

8. The method of claim 7, wherein the said at least one change is the monitoring vehicle moving to within at least a threshold distance from the monitored vehicle, wherein the threshold distance is based at least on a speed of the monitoring vehicle.

9. The method of claim 1, wherein the first visual data and the second visual data are captured by a plurality of cameras mounted on or in the monitoring vehicle.

10. The method of claim 1, wherein the change or changes to an aspect of the monitored vehicle comprises at least one of:

a change in feature size of the monitored vehicle,

a change in an angle of incidence of the monitored vehicle from the monitoring vehicle, or

a change in dimension or shape of a determined bounding box for the monitored vehicle.

11. The method of claim 1, wherein the occurrence of the unsafe driving event is further determined based on at least one of speed data, accelerometer data, gyrometer data, inertial sensor data, atmospheric sensor data, on-board diagnostic (OBD-II) data, or global positioning system (GPS) data.

12. The method of claim 8, wherein the monitoring vehicle is determined to have caused the occurrence of the unsafe driving event.

13. The method of claim 1, further comprising determining, by the at least one processor, a driver grade based at least in part on the determination of the cause of the occurrence of the unsafe driving event.

14. The method of claim 1, further comprising transmitting, by the at least one processor, first data to a second computing device based at least in part on the determination of the cause of the occurrence of the unsafe driving event.

15. The method of claim 14, further comprising determining, by at least one processor of the second computing device, a driver grade based at least in part on the first data.

26. The method of claim 1, wherein the determination whether the monitoring vehicle or the monitored vehicle has caused the occurrence of the unsafe driving event occurs prior to an accident occurring.

16. An apparatus comprising:

a memory;

at least one processor operatively coupled to the memory; and

a first set of instructions stored on the memory and configured to be executed by the at least one processor to cause the at least one processor to;

initially receive first visual data captured by at least one camera associated with a monitoring vehicle;

identify a monitored vehicle from the first visual data;

receive second visual data captured by the at least one camera, wherein the second visual data is captured by the at least one camera subsequent to the capture of the first visual data;

identify the monitored vehicle from the second visual data based at least in part on a spatial location of the monitored vehicle within the first visual data and the second visual data;

determine at least one change to an aspect of the monitored vehicle based on a comparison of the first visual data and the second visual data;

determine an occurrence of an unsafe driving event comprising an unsafe following distance between the monitoring vehicle and the monitored vehicle, wherein the monitoring vehicle is following the monitored vehicle; and

determine whether the monitoring vehicle or the monitored vehicle caused the occurrence of the unsafe driving event based at least in part on said at least one change.

17. The apparatus of claim 16, wherein identification of the monitored vehicle from the visual data further comprises that the at least one processor is configured to:

determine a presence of the monitored vehicle from the first visual data; and

produce one or more first bounding boxes for the monitored vehicle that is determined to be present.

18. The apparatus of claim 17, wherein the at least one processor is further configured to:

analyze the second visual data from the at least one camera based on the one or more first bounding boxes for the monitored vehicle; and

determine one or more second bounding boxes for the monitored vehicle based on the second visual data.

19. The apparatus of claim 18, wherein the change or changes to the aspect of the monitored vehicle is determined based on a comparison of the one or more first bounding boxes to the one or more second bounding boxes for the monitored vehicle.

20. The apparatus of claim 16, wherein the at least one processor is further configured to determine, based on the change or changes to the aspect, a difference in speed between the monitoring vehicle and the monitored vehicle.

21. The apparatus of claim 20, wherein the at least one processor is further configured to determine that the monitoring vehicle or the monitored vehicle is traveling at a speed that is different from surrounding traffic by at least a predetermined threshold, and wherein the occurrence of the unsafe driving event is determined at least in part based on the difference in speed between the monitoring vehicle and the monitored vehicle.

22. A non-transitory computer readable medium having instructions stored thereon that, upon execution by a computing device, cause the computing device to perform operations, wherein the instructions comprise:

instructions to initially receive first visual data captured by at least one camera associated with a monitoring vehicle;

instructions to identify a monitored vehicle from the first visual data;

instructions to receive second visual data captured by the at least one camera, wherein the second visual data is captured by the at least one camera subsequent to the capture of the first visual data;

instructions to identify the monitored vehicle from the second visual data based at least in part on a spatial location of the monitored vehicle within the first visual data and the second visual data;

instructions to determine at least one change to an aspect of the monitored vehicle based on a comparison of the first visual data and the second visual data;

instructions to determine an occurrence of an unsafe driving event comprising an unsafe following distance between the monitoring vehicle and the monitored vehicle, wherein the monitoring vehicle is following the monitored vehicle; and

instructions to determine whether the monitoring vehicle or the monitored vehicle caused the occurrence of the unsafe driving event based at least in part on said at least one change.

23. The non-transitory computer readable medium of claim 22, wherein the instructions further comprise:

instructions to crop the first visual data such that the monitored vehicle identified from the first visual data remains;

instructions to crop the second visual data such that the monitored vehicle identified from the second visual data remains; and

instructions to determine the change or changes to the aspect based on a comparison of the cropped visual data and the cropped second visual data.

24. The non-transitory computer readable medium of claim 23, wherein the instructions further comprise instructions to down sample the first visual data and the second visual data before or after the first visual data and the second visual data are cropped.

25. The non-transitory computer readable medium of claim 22, wherein the instructions further comprise:

instructions to capture a plurality of visual data frames from which the monitored vehicle are identified;

instructions to determine a confidence level for object detection of each of the plurality of visual data frames;

instructions to select a first visual data frame as the first visual data based on the confidence level of the first visual data frame; and

instructions to select a second visual data frame as the second visual data based on the confidence level of the second visual data frame.