Selecting Visible Regions in Nighttime Images for Performing Clear Path Detection

• US 20130202152A1
• Filed: 02/06/2012
• Published: 08/08/2013
• Est. Priority Date: 02/06/2012
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhang, Wende, Wang, Jinsong
• Granted Patent: US 8,948,449 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method provides for determining visible regions in a captured image during a nighttime lighting condition. An image is captured from an image capture device mounted to a vehicle. An intensity histogram of the captured image is generated. An intensity threshold is applied to the intensity histogram for identifying visible candidate regions of a path of travel. The intensity threshold is determined from a training technique that utilizes a plurality of training-based captured images of various scenes. An objective function is used to determine objective function values for each correlating intensity value of each training-based captured image. The objective function values and associated intensity values for each of the training-based captured images are processed for identifying a minimum objective function value and associated optimum intensity threshold for identifying the visible candidate regions of the captured image.

Claims

1. A method of determining visible regions in a captured image during a nighttime lighting condition, the method comprising the steps of:

capturing an image from an image capture device mounted to a vehicle;

generating an intensity histogram of the captured image;

applying an intensity threshold to the intensity histogram for identifying visible candidate regions of a path of travel, the intensity threshold determined from a training technique that utilizes a plurality of training-based captured images of various scenes, wherein an objective function is used to determine objective function values for each correlating intensity value of each training-based captured image, wherein the objective function values and associated intensity values for each of the training-based captured images are processed for identifying a minimum objective function value and associated optimum intensity threshold for identifying the visible candidate regions of the captured image.

2. The method of claim 1 wherein the optimum intensity threshold is determined by one of a plurality of training techniques, wherein a mean squared error is determined for each optimum intensity threshold determined by each training technique, wherein the optimum intensity threshold associated with the lowest determined mean squared error is selected for identifying the candidate regions in the captured image obtained by the image capture device.

3. The method of claim 2 wherein a fixed intensity threshold technique is applied for determining the optimum intensity threshold, the fixed intensity threshold technique comprises:

summing the objective function values for each correlating intensity value of each training-based captured image;

identifying the minimum objective function value from the summed objective function values;

identifying the intensity threshold associated with the identified minimum objective function value; and

applying the identified intensity threshold to the intensity histogram for identifying pixels associated with the candidate regions of the captured image.

4. The method of claim 2 wherein a fixed intensity percentage technique is applied for determining the optimum intensity threshold, the fixed intensity-percentage threshold technique comprises:

determining a cumulative intensity histogram for each training-based captured image;

determining an objective function value for each intensity value in each training-based captured image;

generating a cumulative histogram for each training-based capture image, each cumulative histogram being generated as a function of a percentage of the intensity values in the training-based captured image;

generating an objective function value-percentage-based intensity value correlation graph for each training-based capture image that identifies each percentage-based intensity value in each image to a respective objective function value;

averaging the objective function values for each percentage-based intensity value;

identifying the minimum objective function value from the averaged objective function values;

identifying the respective percentage-based intensity value associated with the minimum objective function value;

identifying the intensity value threshold associated with the identified percentage-based intensity value; and

applying the identified intensity threshold to the intensity histogram of the captured image for identifying pixels associated with the candidate regions of the captured image.

5. The method of claim 2 wherein an intensity threshold regression technique is applied for determining the optimum intensity threshold, the intensity threshold regression technique comprises:

generating a cumulative histogram of the captured image;

applying a regression modeling technique for determining the intensity threshold from the cumulative histogram of the captured image;

applying the identified intensity threshold to the intensity histogram for identifying pixels associated with the candidate regions of the captured image.

6. The method of claim 5 wherein the regression modeling technique applied for identifying the intensity threshold includes a least squared regression approach.

7. The method of claim 5 wherein the regression modeling technique applied for identifying the intensity threshold includes a robust regression approach.

8. The method of claim 5 wherein the regression modeling technique applied for identifying the optimum intensity threshold is determined in response to determining a mean squared error for each regression modeling technique as applied to each training-based captured image and in response to identifying the respective regression modeling technique producing a lowest mean square between the compared regression modeling techniques.

9. The method of claim 5 wherein the regression modeling technique includes a linear regression modeling technique and is represented by the following formula:

10. The method of claim 2 wherein a intensity percentage regression technique is applied for determining the intensity threshold, the intensity percentage regression technique comprises:

generating a cumulative histogram of the captured image;

applying a regression modeling technique for determining a percentage-based intensity value;

identifying the intensity threshold associated with the identified percentage-based intensity value;

applying the identified intensity threshold to the intensity histogram for identifying pixels associated with the candidate regions of the captured image.

11. The method of claim 10 wherein the regression modeling technique applied for identifying the intensity threshold includes a least square regression approach.

12. The method of claim 10 wherein the regression modeling technique applied for identifying the intensity threshold includes a robust regression approach.

13. The method of claim 10 wherein the regression modeling technique applied for identifying the intensity threshold is determined in response to determining a mean squared error for each regression modeling technique as applied to each training-based captured image and in response to identifying the respective regression modeling technique producing a lowest mean square error between the compared regression modeling techniques.

14. The method of claim 5 wherein the regression modeling technique is represented by the following formula:

15. The method of claim 1 wherein the objective function is derived from the training technique using observations of sampled nighttime captured images.

16. The method of claim 15 wherein observations include ground-truth labeling used to identify visible and invisible regions of the sampled nighttime images, the ground-truth labeling includes observations relating to a labeled road that is expected to be visible within a region of interest in the sampled nighttime captured images, observations relating to an unlabeled region that are expected to be in an invisible region within the sampled nighttime captured images, and observations relating to an invisible region that are expected to contain no textures within the sampled nighttime captured images.

17. The method of claim 16 wherein the objective function is represented by the following formula:

ƒ

(x)=w₁·

(1−

rd_vis(x))+w₂·

(1−

unL_vis(x))+w₃·

grdmag_invis(x)

where x is the intensity threshold of the visible region of interest in one sampled nighttime captured image, rd_vis is the ratio of labeled road areas in a visible region of interest in the sampled captured images over a total labeled region in the sampled nighttime captured images, unL_vis is the ratio of unlabeled areas classified as being invisible over a total unlabeled region area in the sampled nighttime captured images, grdmag_invis is a sum of a gradient magnitude in the invisible region, and w is the weights of each component in the objective function.

18. The method of claim 1 wherein morphological smoothing is applied to the candidate regions to reduce noise and establish a smooth region.

19. The method of claim 1 wherein the application for assisting a driver in determining a path includes a front view camera and an output display device.

20. The method of claim 1 wherein the application for assisting a driver in determining a path includes an object detection sensing system.

21. The method of claim 1 wherein illumination normalization is applied to the captured image for contrasting regions of the image for enhancing image classification.