Computer vision based driver assistance devices, systems, methods and associated computer executable code

• US 10,007,854 B2
• Filed: 07/07/2016
• Issued: 06/26/2018
• Est. Priority Date: 07/07/2016
• Status: Expired due to Fees

• Current Assignee: KAMI Vision Incorporated
• Original Assignee: ANTS TECHNOLOGY (HK) LIMITED
• Inventors: Blayvas, Ilya, Nosko, Pavel, Arie Nachimson, Mica, Dror, Oranit, Fridental, Ron, Gvirzer, Ophir, Perets, Gal, Rosen, Alex, Simakov, Denis, Zeldin, Masha
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/203,881
• Publication Number:
  US 20180012085A1
• Time in Patent Office: 719 Days
• Field of Search: 348 36, 348 37, 348 35
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/50 Constructional details
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The present invention includes computer vision based driver assistance devices, systems, methods and associated computer executable code (hereinafter collectively referred to as: “ADAS”). According to some embodiments, an ADAS may include one or more fixed image/video sensors and one or more adjustable or otherwise movable image/video sensors, characterized by different dimensions of fields of view. According to some embodiments of the present invention, an ADAS may include improved image processing. According to some embodiments, an ADAS may also include one or more sensors adapted to monitor/sense an interior of the vehicle and/or the persons within. An ADAS may include one or more sensors adapted to detect parameters relating to the driver of the vehicle and processing circuitry adapted to assess mental conditions/alertness of the driver and directions of driver gaze. These may be used to modify ADAS operation/thresholds.

Claims

1. A system for computer vision based driver assistance, said system comprising:

an adjustable camera having an angle of view of 75 degrees or greater and adapted to be adjustably mounted to a vehicle, such that the orientation of said camera in relation to the vehicle is adjustable;

one or more fixed cameras, having an angle of view of 70 degrees or less and adapted to be mounted to a vehicle, such that the orientation of said camera in relation to the vehicle is fixed;

first processing circuitry communicatively coupled to said adjustable and fixed cameras and adapted to process images captured by said adjustable and fixed cameras to identify hazardous situations relating to the vehicle;

one or more driver monitors comprising interior facing image sensors adapted to be positioned within a vehicle and sense a direction of gaze of a driver of the vehicle;

second processing circuitry communicatively coupled to said first processing circuitry and said driver monitors and adapted to;

(i) determine a level of risk for each identified hazardous situation, and

(ii) issue a warning or intervene in vehicle control, upon determining a level of risk associated with a given hazardous situation is above a dynamic threshold;

wherein the dynamic threshold is dependent upon a current direction of gaze of the driver sensed by said driver monitor.

2. The system according to claim 1, wherein said one or more fixed cameras comprise at least two cameras having an angle of view of 70 degrees or less.

3. The system according to claim 2, wherein said at least two cameras capture stereo images of an area in front of the vehicle and said first processing circuitry is further adapted to derive depth information relating to objects appearing in the images based on the stereo images.

4. The system according to claim 2, wherein said first processing circuitry is further adapted to compensate for view obstructions of one of the at least two cameras using image data captured by a different camera of the at least two cameras.

5. The system according to claim 1, wherein said first processing circuitry is further adapted to use said fixed cameras to calibrate said adjustable camera.

6. The system according to claim 1, wherein said adjustable camera is adapted to be mounted on a rear view mirror of the vehicle.

7. The system according to claim 1, wherein said fixed cameras are adapted to be mounted on a windshield of the vehicle.

8. The system according to claim 1, wherein said fixed cameras are adapted to be mounted adjacent to headlights of the vehicle.

9. The system according to claim 1, further comprising communication circuitry adapted to communicate over a distributed data network, wherein said first processing circuitry is adapted to utilize a neural network to perform at least a portion of the image processing.

10. The system according to claim 1, wherein said adjustable camera is adapted to be mounted on a rear view mirror of the vehicle and said fixed cameras include at least two cameras adapted to be mounted on a windshield of the vehicle.

11. The system according to claim 1, wherein said second processing circuitry is further adapted to measure a lapse of time between each:

(a) occurrence of one or more given hazardous situations, of the identified hazardous situations, and (b) a turn of the direction of gaze of the driver towards the given hazardous situation.

12. A system for computer vision based driver support, said system comprising:

one or more exterior facing image sensors adapted to be mounted to a vehicle, such as to capture images of an area in front of the vehicle;

one or more driver monitors comprising interior facing image sensors adapted to be positioned within a vehicle and sense a direction of gaze of a driver of the vehicle;

processing circuitry communicatively coupled to said exterior facing image sensors and said driver monitors and adapted to;

(i) process images captured by said exterior facing image sensors to identify hazardous situations relating to the vehicle;

(ii) determine a level of risk for each identified hazardous situation, and

(iii) issue a warning or intervene in vehicle control, upon determining a level of risk associated with a given hazardous situation is above a dynamic threshold;

wherein the dynamic threshold is dependent upon a current direction of gaze of the driver sensed by said driver monitor.

13. The system according to claim 12, wherein said processing circuitry is further adapted to measure a lapse of time between each:

(a) occurrence of one or more given hazardous situations, of the identified hazardous situations, and (b) a turn of the direction of gaze of the driver towards the given hazardous situation.

14. The system according to claim 13, wherein said processing circuitry is further adapted to assess an alertness of the driver based on the measured lapses of time.

15. The system according to claim 13, wherein said system is further adapted to:

a. dynamically create, for each of a set of individual users, reference time lapse data, between occurrences of hazardous situations and turn of gaze towards the hazardous situations, by recording reference time lapse data for each of the individual users; and

b. compare time lapse data of current identified hazardous situations to reference time lapse data for a current driver, to thereby assess an alertness of the current driver.

16. The system according to claim 12, further comprising sensors adapted to sense positions of the driver of the vehicle and passengers within the vehicle.

17. The system according to claim 16, wherein said processing circuitry is further adapted to manipulate controls of the vehicle to alter a point of impact to thereby mitigate injury to the driver and passengers based on the sensed positions of the driver and passengers.

18. A system for computer vision based driver support, said system comprising:

one or more image sensors adapted to be mounted to a vehicle;

such as to capture images of an area in front of the vehicle;

one or more interior sensors adapted to be positioned within a vehicle and sense parameters relating to an alertness of a driver of the vehicle;

processing circuitry communicatively coupled to said image sensors and said interior sensors and adapted to;

(i) process images captured by said image sensors to identify hazardous situations relating to the vehicle;

(ii) determine a level of risk for each identified hazardous situation, and

(iii) issue a warning or intervene in vehicle control, upon determining a level of risk associated with a given hazardous situation is above a dynamic threshold;

wherein the dynamic threshold is dependent upon a current alertness of the driver determined by the processing circuitry based upon the parameters sensed by said interior sensors relating to an alertness of a driver of the vehicle.

19. The system according to claim 18, wherein said processing circuitry is further adapted to receive signals from media devices within the vehicle, indicating a level of interaction of the driver with the media devices, and factor a current level of interaction indicated by the media devices when assessing a current alertness of the driver.