DRIVER BEHAVIOR MONITORING

• US 20170200061A1
• Filed: 02/21/2017
• Published: 07/13/2017
• Est. Priority Date: 01/11/2016
• Status: Active Grant

• Current Assignee: Netradyne Inc.
• Original Assignee: Netradyne Inc.
• Inventors: Agrawal, Avneesh, Julian, David Jonathan
• Granted Patent: US 10,460,600 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G07C 5/0808 Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 for classifying traffic situation
• CPC Class Codes: G08G 1/0137 for specific applications
• CPC Class Codes: G08G 1/015 with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/052 with provision for determining speed or overspeed speed measuring in general G01P

Abstract

Systems and methods provide, implement, and use using a computer-vision based methods of context-sensitive monitoring and characterization of driver behavior. Additional systems and methods are provided for unsupervised learning of action values, monitoring of a driver'"'"'s environment, and transmitting visual information from a client to a server.

Claims

1. A method comprising:

receiving, by at least one processor of a computing device, visual data from a camera of a first vehicle;

identifying, by the at least one processor, an object or objects from the visual data;

receiving, by the at least one processor, subsequent visual data from the camera;

identifying, by the at least one processor, the object or objects from the subsequent visual data;

determining, by the at least one processor, a change or changes to an aspect of the object or objects based on a comparison of the visual data and the subsequent visual data; and

determining, by the at least one processor, an unsafe driving behavior based at least in part on the change or changes to the aspect of the object or objects.

2. The method of claim 1, wherein the object or objects identified comprise at least one traffic control device.

3. The method of claim 2, wherein the traffic control device is a traffic light, and the unsafe driving behavior comprises the first vehicle running a red light.

4. The method of claim 2, wherein the traffic control device is a stop sign, and the unsafe driving behavior comprises the first vehicle failing to come to a complete stop at a stop sign.

5. The method of claim 2, wherein the traffic control device is a lane marker, and the unsafe driving behavior comprises an unsafe lane change of the first vehicle.

6. The method of claim 1, wherein the object or objects identified comprise at least one other vehicle.

7. The method of claim 6, wherein the unsafe driving behavior comprises the first vehicle tailgating a second vehicle.

8. The method of claim 6, wherein the unsafe driving behavior comprises at least one of:

the first vehicle driving near to a second vehicle for at least a predetermined threshold amount of time, and

the first vehicle driving within at least a threshold distance from the second vehicle, wherein the threshold distance is based on a speed of the first vehicle.

9. The method of claim 1, wherein the visual data is captured by one or more cameras mounted on or in the first vehicle.

10. The method of claim 1, wherein the change or changes to an aspect of the identified object or objects comprises at least one of:

a change in a distance of the identified object or objects from a specified point on the vehicle or a point from which the visual data is captured,

a change in a color of the identified object or objects,

a change in feature size of the identified object or objects,

a change in an angle of incidence of the identified object or objects from the vehicle, and

a change in dimension or shape of a determined bounding box for the identified object or objects.

11. The method of claim 1, wherein the unsafe driving behavior is further determined based on at least one of speed data, accelerometer data, gyrometer data, inertial sensor data, atmospheric sensor data, on-board diagnostic (OBD-II) data, and global positioning system (GPS) data.

12. The method of claim 1, further comprising determining, by the at least one processor, a cause of the determined unsafe driving behavior based at least in part on the change or changes to the aspect of the object or objects.

13. The method of claim 1, further comprising determining, by the at least one processor, a driver grade based at least in part on the determined unsafe driving behavior.

14. The method of claim 1, further comprising transmitting, by the at least one processor, a first data to a second computing device based at least in part on the determined unsafe driving behavior.

15. The method of claim 14, further comprising determining, by at least one processor of the second computing device, a driver grade based at least in part on the first data.

16. An apparatus comprising:

a memory;

at least one processor operatively coupled to the memory; and

a first set of instructions stored on the memory and configured to be executed by the at least one processor to cause the at least one processor to;

receive visual data from a camera of a first vehicle;

identify an object or objects from the visual data;

receive subsequent visual data from the camera;

identify the object or objects from the subsequent visual data;

determine a change or changes to an aspect of the object or objects based on a comparison of the visual data and the subsequent visual data; and

determine an unsafe driving behavior based at least in part on the change or changes to the aspect of the object or objects.

17. The apparatus of claim 16, wherein identification of the object or objects from the visual data further comprises that the at least one processor is configured to:

determine a presence of the object or objects from the visual data; and

produce one or more bounding boxes for the object or objects that were determined to be present.

18. The apparatus of claim 17, wherein the at least one processor is further configured to:

analyze the subsequent visual data from the camera based on the one or more bounding boxes for the object or objects; and

determine subsequent one or more bounding boxes for the object or objects based on the subsequent visual data.

19. The apparatus of claim 17, wherein the change or changes to the aspect of the object or objects is determined based on a comparison of the one or more bounding boxes to the subsequent one or more bounding boxes for the object or objects.

20. The apparatus of claim 16, wherein the visual data is captured at a first time and the subsequent visual data is captured at a second time after the first time.

21. The apparatus of claim 16, wherein the at least one processor is further configured to determine, based on the change or changes to the aspect, a difference in speed between the first vehicle and a second vehicle.

22. The apparatus of claim 21, wherein the unsafe driving behavior comprises the first vehicle traveling at a speed that is different from surrounding traffic by at least a predetermined threshold, and wherein the unsafe driving behavior is determined at least in part by the difference in speed between the first vehicle and the second vehicle.

23. A non-transitory computer readable medium having instructions stored thereon that, upon execution by a computing device, cause the computing device to perform operations, wherein the instructions comprise:

instructions to receive visual data from a camera of a first vehicle;

instructions to identify an object or objects from the visual data;

instructions to receive subsequent visual data from the camera;

instructions to identify the object or objects from the subsequent visual data;

instructions to determine a change or changes to an aspect of the object or objects based on a comparison of the visual data and the subsequent visual data; and

instructions to determine an unsafe driving behavior based at least in part on the change or changes to the aspect of the object or objects.

24. The non-transitory computer readable medium of claim 23, wherein the instructions further comprise:

instructions to crop the visual data such that the object or objects identified from the visual data remain;

instructions to crop the subsequent visual data such that the object or objects identified from the subsequent visual data remain; and

instructions to determine the change or changes to the aspect based on a comparison of the cropped visual data and the cropped subsequent visual data.

25. The non-transitory computer readable medium of claim 24, wherein the instructions further comprise instructions to down sample the visual data and the subsequent visual data before or after the visual data and the subsequent visual data are cropped.

26. The non-transitory computer readable medium of claim 23, wherein the instructions further comprise:

instructions to capture a plurality of visual data frames from which the object or objects are identified;

instructions to determine a confidence level for object detection of each of the plurality of visual data frames;

instructions to select a first visual data frame as the visual data based on the confidence level of the first visual data frame; and

instructions to select a second visual data frame as the subsequent visual data based on the confidence level of the second visual data frame.