Enhanced camera object detection for automated vehicles

• US 10,366,310 B2
• Filed: 08/18/2017
• Issued: 07/30/2019
• Est. Priority Date: 09/12/2016
• Status: Active Grant

• Current Assignee: Aptiv Technologies AG
• Original Assignee: Aptiv Technologies Limited (BorgWarner Incorporated)
• Inventors: Zermas, Dimitris, Izzat, Izzat H., Mangalgiri, Anuradha
• Primary Examiner(s): Tsai, Tsung Yin
• Application Number:
  US15/680,854
• Publication Number:
  US 20180075320A1
• Time in Patent Office: 711 Days
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/16 Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08

Abstract

An illustrative example object detection system includes a camera having a field of view. The camera provides an output comprising information regarding potential objects within the field of view. A processor is configured to select a portion of the camera output based on information from at least one other type of detector that indicates a potential object in the selected portion. The processor determines an Objectness of the selected portion based on information in the camera output regarding the selected portion.

Claims

1. An object detection system, comprising:

a camera having a field of view, the camera providing an output comprising information regarding potential objects within the field of view; and

a processor that is configured to select a portion of the camera output based on information from at least one other type of detector that indicates a potential object in the selected portion, the processor being configured to ignore other portions of the camera output that, based on the information from the at least one other type of detector, do not include a potential object, the processor determining an Objectness of the selected portion based on information in the camera output regarding the selected portion.

2. The object detection system of claim 1, wherein the processor is configured to

locate a plurality of segments in the selected portion;

divide at least one of the segments into patches; and

determine the Objectness of each of the patches, respectively.

3. The object detection system of claim 2, wherein

the processor is configured to determine a total Objectness of an entire at least one of the segments;

and

the processor is configured to determine the Objectness of the at least one of the segments based on the Objectness of each of the patches and the total Objectness.

4. The object detection system of claim 1, wherein

the processor is configured to divide up the selected portion into segments;

and

the processor is configured to arrange the segments based on an Objectness of the respective segments.

5. The object detection system of claim 4, wherein the segments include at least one segment having a first geometry and at least one other segment having a second, different geometry.

6. The object detection system of claim 5, wherein at least the first geometry corresponds to a distribution of data points from the at least one other type of detector within the at least one segment.

7. The object detection system of claim 1, comprising the at least one other type of detector and wherein the at least one other type of detector comprises one of a radar detector or a LIDAR detector.

8. The object detection system of claim 1, wherein

the processor is configured to recognize a clustered set of data points from the at least one other type of detector;

and

the processor is configured to select at least one clustered set of data points as the selected portion.

9. The object detection system of claim 1, wherein

the at least one other type of detector provides a LIDAR output having an intensity;

and

the processor determines the Objectness from at least one of the camera output and the intensity of the LIDAR output.

10. The object detection system of claim 1, wherein

the camera output comprises a plurality of images;

the plurality of images are in a time-based sequence;

the processor is configured to use the plurality of images to determine motion cues corresponding to movement of a potential object in the selected portion; and

the processor is configured to use the motion cues when determining the Objectness of the selected portion.

11. The object detection system of claim 1, wherein

the processor is configured to determine a respective Objectness of a plurality of segments of the selected portion;

the processor is configured to rank the Objectness of each of the segments; and

the processor is configured to select a highest ranked Objectness to identify a location of a potential object.

12. The object detection system of claim 1, wherein the processor is configured to provide an object location estimation within an identified area of the selected portion of the camera output.

13. A method of detecting at least one potential object, the method comprising:

selecting a portion of a camera output based on information from at least one other type of detector that indicates a potential object in the selected portion;

ignoring other portions of the camera output that, based on the information from the at least one other type of detector, do not include a potential object; and

determining an Objectness of the selected portion based on information in the camera output regarding the selected portion.

14. The method of claim 13, comprising

dividing up the selected portion into segments;

and

determining the Objectness of each of the segments, respectively.

15. The method of claim 14, comprising

dividing at least one of the segments into a plurality of patches;

determining a total Objectness of the entire at least one of the segments;

determining an Objectness of each of the patches; and

determining the Objectness of the at least one of the segments based on the Objectness of each of the patches and the total Objectness.

16. The method of claim 13, wherein

dividing up the selected portion into segments comprises configuring respective geometries of the segments based on information from the at least one other type of detector;

the segments include at least one segment having a first geometry and at least one other segment having a second, different geometry; and

at least the first geometry corresponds to a distribution of data points from the at least one other type of detector within the at least one segment.

17. The method of claim 13, wherein selecting the portion of the camera output comprises recognizing a clustered set of data points from the at least one other type of detector.

18. The method of claim 13, wherein the camera output comprises a plurality of images in a time-based sequence and the method comprises

using the plurality of images to determine movement cues of a potential object in the selected portion;

and

using the movement cues to determine the Objectness of the selected portion.

19. The method of claim 13, comprising

determining a respective Objectness of a plurality of segments of the selected portion;

ranking the Objectness of each of the segments; and

selecting a highest ranked Objectness to identify a location of a potential object.

20. The method of claim 13, comprising providing an object location estimation within an identified area of the selected portion of the camera output.

21. The method of claim 13, wherein

the at least one other type of detector provides a LIDAR output having an intensity;

and

determining the Objectness comprises using at least one of the camera output and the intensity of the LIDAR output.