Embedded surround vision-based driver assistance for safe zone estimation

• US 10,417,506 B2
• Filed: 11/18/2016
• Issued: 09/17/2019
• Est. Priority Date: 11/19/2015
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Satzoda, Ravi Kumar, Trivedi, Mohan M.
• Primary Examiner(s): Patel, Jayanti K
• Assistant Examiner(s): Haghani, Shadan E
• Application Number:
  US15/356,495
• Publication Number:
  US 20170147891A1
• Time in Patent Office: 1,033 Days
• Field of Search: None
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G06V 10/22 by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A driver assistance system including at least a first camera. The first camera can be configured to obtain image data of a region external to the host vehicle. Lane detection analysis can be performed on the image data to identify a region of interest in the image data. The region of interest can be divided into portions. Each portion can be analyzed to determine whether it includes the characteristics of an under-vehicle region. When an under-vehicle region is detected a portion of the image adjacent an under-vehicle region can be analyzed to determine whether it includes an image of another vehicle. A distance between the another vehicle and the host vehicle can be determined using the image data. A threat analysis can be performed to provide a level of threat posed by the another vehicle to the host vehicle.

Claims

1. A system comprising:

a first device comprising;

at least one forward-facing camera configured to at least obtain image data of a first plurality of regions external to a front of a host vehicle;

a second device comprising;

at least one rearward-facing camera configured to obtain image data of a second plurality of regions external to a rear of the host vehicle;

the second device is configured to perform image analysis on the image data obtained by the rearward-facing camera and determine a threat to the host vehicle by another vehicle in at least one region in the second plurality of regions; and

transmit determined threat data from the second device to the first device;

at least one processor embedded in the first device; and

at least one memory configured to at least store computer-readable instructions, which, when processed by the at least one processor, cause the at least one processor to perform one or more operations, the one or more operations comprising;

receiving, from the at least one forward-facing camera, the image data of the first plurality of regions and the second plurality of regions and determining at least one region external to the host vehicle having a likelihood of including another vehicle;

receiving a motion information associated with the host vehicle;

determining, in an image domain of the image data associated with the at least one determined region, a location of a lane in which the host vehicle is travelling;

analyzing a first portion of the image data associated with the at least one determined region that is within the lane in which the host vehicle is travelling, the analyzing of the first portion to determine whether the first portion of the image data associated with the at least one determined region contains an image of an under-vehicle region, the analyzing including executing a machine learning analysis of one or more grey color levels of the first portion in relation to one or more grey color level of a road surface of the lane in which the host vehicle is travelling using historical values of grey color levels associated with one or more road surfaces and one or more vehicles;

analyzing, in response to determining that the first portion of the image data associated with the at least one determined region contains the image of the under-vehicle region, a second portion of the image data to determine whether the second portion of the image data associated with the at least one determined region contains an image of another vehicle, the second portion of the image data being adjacent to the first portion of the image data associated with the at least one determined region; and

determining, in response to determining that the second portion of the image data associated with the at least one determined region contains the image of another vehicle, the received threat data from the second device, and the received motion information associated with the host vehicle, a threat posed by the another vehicle to the host vehicle.

2. The system of claim 1, wherein the analyzing of the first portion of the image data comprises:

generating, in the image domain of the image data, a region of interest, the region of interest being within the lane in which the host vehicle is travelling;

dividing the region of interest into a set of discrete blocks; and

analyzing the set of discrete blocks to determine whether a block of the set of discrete blocks contains the image of the under-vehicle region.

3. The system of claim 2, wherein the analyzing of the second portion of the image comprises:

bounding, in response to determining that the block of the set of discrete blocks contains the image of the under-vehicle region, a portion of the image data adjacent to the block of the set of discrete blocks; and

analyzing the bounded portion of the image data to determine whether the bounded portion of the image data contains the image of the another vehicle.

4. The system of claim 3, wherein the determining of the threat posed by the another vehicle to the host vehicle, comprises:

determining, based on a location in the image domain of the image data of the bounded portion of the image data, a distance between the host vehicle and the another vehicle.

5. The system of claim 1, wherein the determining of the threat posed by the another vehicle to the host vehicle, comprises:

determining, based on the motion information associated with the host vehicle, a motion of the another vehicle relative to the host vehicle.

6. The system of claim 1, wherein the operations further comprise:

generating an alert to a driver of the host vehicle in response to the determining that the threat posed by the another vehicle is above a threshold threat value.

7. The system of claim 1, wherein analyzing the first portion of the image data to determine whether the first portion of the image data contains the image of the under-vehicle region includes at least applying a probabilistic distribution curve providing an indication of a likelihood that the first portion of the image data contains the image of the under-vehicle region.

8. The system of claim 1, wherein the operations further comprise:

generating a second region of interest associated with an adjacent lane to the lane in which the host vehicle is travelling, the second region of interest generated based on a lane offset value and calibration information associated with forward-facing camera.

9. The system of claim 1, wherein the second portion of the image data is above the first portion of the image data.

10. A method comprising:

receiving, from at least one forward-facing camera, an image data of a first plurality of regions and a second plurality of regions and determining at least one region external to a front of a host vehicle having a likelihood of including another vehicle;

receiving a motion information associated with the host vehicle;

determining, in an image domain of the image data associated with the at least one determined region, a location of a lane in which the host vehicle is travelling;

analyzing a first portion of the image data associated with the at least one determined region that is within the lane in which the host vehicle is travelling, the analyzing of the first portion to determine whether the first portion of the image data associated with the at least one determined region contains an image of an under-vehicle region, the analyzing including executing a machine learning analysis of one or more grey color levels of the first portion in relation to one or more grey color level of a road surface of the lane in which the host vehicle is travelling using historical values of grey color levels associated with one or more road surfaces and one or more vehicles;

analyzing, in response to determining that the first portion of the image data associated with the at least one determined region contains the image of the under-vehicle region, a second portion of the image data to determine whether the second portion of the image data associated with the at least one determined region contains an image of another vehicle, the second portion of the image data being adjacent to the first portion of the image data associated with the at least one determined region;

obtaining image data of another region external to a rear of the host vehicle at least one rearward-facing camera;

performing image analysis on the image data obtained by the rearward-facing camera and determining a threat to the host vehicle by another vehicle in at least one region in the second plurality of regions;

receiving the determined threat data; and

determining, in response to determining that the second portion of the image data associated with the at least one determined region contains the image of another vehicle, the received threat data from the second device, and the received motion information associated with the host vehicle, a threat posed by the another vehicle to the host vehicle.

11. The method of claim 10, wherein the analyzing of the first portion of the image data comprises:

generating, in the image domain of the image data, a region of interest, the region of interest being within the lane in which the host vehicle is travelling;

dividing the region of interest into a set of discrete blocks; and

analyzing the set of discrete blocks to determine whether a block of the set of discrete blocks contains the image of the under-vehicle region.

12. The method of claim 11, wherein the analyzing of the second portion of the image comprises:

bounding, in response to determining that the block of the set of discrete blocks contains the image of the under-vehicle region, a portion of the image data adjacent to the block of the set of discrete blocks; and

analyzing the bounded portion of the image data to determine whether the bounded portion of the image data contains the image of the another vehicle.

13. The method of claim 12, wherein the determining of the threat posed by the another vehicle to the host vehicle, comprises:

determining, based on a location in the image domain of the image data of the bounded portion of the image data, a distance between the host vehicle and the another vehicle.

14. The method of claim 10, wherein the determining of the threat posed by the another vehicle to the host vehicle, comprises:

determining, based on the motion information associated with the host vehicle, a motion of the another vehicle relative to the host vehicle.

15. The method of claim 10, further comprising:

generating an alert to a driver of the host vehicle in response to the determining that the threat posed by the another vehicle is above a threshold threat value.

16. The method of claim 10, wherein analyzing the first portion of the image data to determine whether the first portion of the image data contains the image of the under-vehicle region includes at least applying a probabilistic distribution curve providing an indication of a likelihood that the first portion of the image data contains the image of the under-vehicle region.

17. The method of claim 10, further comprising:

generating a second region of interest associated with an adjacent lane to the lane in which the host vehicle is travelling, the second region of interest generated based on a lane offset value and calibration information associated with forward-facing camera.

18. The method of claim 10, wherein the second portion of the image data is above the first portion of the image data.

19. A non-transient computer readable medium containing program instructions which, when executed by at least one processor, cause the at least one processor to perform one or more operations, the operations comprising:

receiving, from at least one forward-facing camera, an image data of a first plurality of regions and a second plurality of regions and determining at least one region external to a front of a host vehicle having a likelihood of including another vehicle;

receiving a motion information associated with the host vehicle;

determining, in an image domain of the image data associated with the at least one determined region, a location of a lane in which the host vehicle is travelling;

analyzing a first portion of the image data associated with the at least one determined region that is within the lane in which the host vehicle is travelling, the analyzing of the first portion to determine whether the first portion of the image data associated with the at least one determined region contains an image of an under-vehicle region, the analyzing including executing a machine learning analysis of one or more grey color levels of the first portion in relation to one or more grey color level of a road surface of the lane in which the host vehicle is travelling using historical values of grey color levels associated with one or more road surfaces and one or more vehicles;

analyzing, in response to determining that the first portion of the image data associated with the at least one determined region contains the image of the under-vehicle region, a second portion of the image data to determine whether the second portion of the image data associated with the at least one determined region contains an image of another vehicle, the second portion of the image data being adjacent to the first portion of the image data associated with the at least one determined region;

obtaining image data of another region external to a rear of the host vehicle at least one rearward-facing camera;

performing image analysis on the image data obtained by the rearward-facing camera and determining a threat to the host vehicle by another vehicle in at least one region in the second plurality of regions;

receiving the determined threat data; and

determining, in response to determining that the second portion of the image data associated with the at least one determined region contains the image of another vehicle, and the received threat data from the second device, and the received motion information associated with the host vehicle, a threat posed by the another vehicle to the host vehicle.

20. A system comprising:

at least one processor configured to receive an image data of a first plurality of regions and a second plurality of regions and determining at least one region external to a front of a host vehicle having a likelihood of including another vehicle from at least one forward-facing camera;

at least one processor configured to receive a motion information associated with the host vehicle;

at least one processor configured to determine, in an image domain of the image data associated with the at least one determined region, a location of a lane in which the host vehicle is travelling;

at least one processor configured to analyze a first portion of the image data associated with the at least one determined region that is within the lane in which the host vehicle is travelling, the analyzing of the first portion to determine whether the first portion of the image data associated with the at least one determined region contains an image of an under-vehicle region, the analyzing including executing a machine learning analysis of one or more grey color levels of the first portion in relation to one or more grey color level of a road surface of the lane in which the host vehicle is travelling using historical values of grey color levels associated with one or more road surfaces and one or more vehicles;

at least one processor configured to analyze, in response to determining that the first portion of the image data associated with the at least one determined region contains the image of the under-vehicle region, a second portion of the image data to determine whether the second portion of the image data associated with the at least one determined region contains an image of another vehicle, the second portion of the image data being adjacent to the first portion of the image data associated with the at least one determined region;

at least one processor configured to obtain image data of another region external to a rear of the host vehicle at least one rearward-facing camera;

at least one processor configured to perform image analysis on the image data obtained by the rearward-facing camera and determining a threat to the host vehicle by another vehicle in at least one region in the second plurality of regions;

at least one processor configured to receive the determined threat data; and

at least one processor configured to determine, in response to determining that the second portion of the image data associated with the at least one determined region contains the image of another vehicle, the received threat data from the second device, and the received motion information associated with the host vehicle, a threat posed by the another vehicle to the host vehicle.