Method and apparatus for enhancing the dynamic range of a stereo vision system

• US 7,561,731 B2
• Filed: 12/27/2004
• Issued: 07/14/2009
• Est. Priority Date: 12/27/2004
• Status: Expired due to Fees

• Current Assignee: TRW Automotive USLLC (Zeppelin - Stiftung der Stadt Friedrichshafen)
• Original Assignee: TRW Automotive USLLC (Zeppelin - Stiftung der Stadt Friedrichshafen)
• Inventors: Luo, Yun, Wallace, Jon K.
• Primary Examiner(s): Bella; Matthew C
• Assistant Examiner(s): Newman; Michael A
• Application Number:
  US11/022,723
• Publication Number:
  US 20060140510A1
• Time in Patent Office: 1,660 Days
• Field of Search: 382/106, 382/154, 382/274, 382/299
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods are provided for producing a stereo disparity map representing an imaged subject. A plurality of sensors (102 and 104) are operative to produce a plurality of images. Each image has an associated exposure according to an exposure level of an associated sensor. A system control (106) determines exposure levels for the plurality of sensors according to an exposure selection algorithm. A disparity processing component (112) is operative to generate a stereo disparity map from a plurality of images. An image merger (110) is operative to merge a plurality of images to produce a composite image having an increased dynamic range.

Claims

1. A system for producing a stereo disparity map representing an imaged subject, the system comprising:

a plurality of sensors configured to produce a first plurality of images according to a first exposure level for the plurality of sensors;

a disparity processing component that is configured to generate a first stereo disparity map from the first plurality of images;

a system control configured to select a second exposure level for the plurality of sensors according to an exposure selection algorithm if the stereo disparity map contains an unacceptable amount of dark or saturated pixels, direct the plurality of sensors to produce a second plurality of images, and direct the disparity processing component to generate a second stereo disparity map from the second plurality of images; and

an image merger that is configured to merge the first stereo disparity map and the second stereo disparity map, the merger of the first stereo disparity map and the second stereo disparity map being performed as to produce a composite stereo disparity map having an increased dynamic range.

2. The system of claim 1 wherein the image merger is operative to provide the composite image to the system control, and the system control configured to select a third exposure level for the plurality of sensors according to the exposure selection algorithm if the composite image contains an unacceptable amount of dark or saturated pixels, direct the plurality of sensors to produce a third plurality of images, and direct the disparity processing component to generate a third stereo disparity map from the second plurality of images, the image merger being configured to merge the third stereo disparity map and the composite stereo disparity map to produce a new composite stereo disparity map having an increased dynamic range.

3. The system of claim 1 wherein the image merger combines the first stereo disparity map and the second stereo disparity map by summing the intensity values of corresponding pixels across the the first stereo disparity map and the second stereo disparity map.

4. The system of claim 3, wherein the image merger sums corresponding pixels only within a region of interest within the the first stereo disparity map and the second stereo disparity map.

5. An actuatable vehicle occupant restraint system comprising the system of claim 1.

6. The system of claim 1, the system further comprising an image analysis component that evaluates the plurality of images generated by the sensors and provides statistics relating to at least one of the plurality of images to the system control.

7. A method for producing a stereo disparity map having an enhanced dynamic range, comprising:

determining a first exposure level for a first image set according to an exposure selection algorithm;

generating the first image set at the determined first exposure level;

generating a first stereo disparity map from the first image set, the first stereo disparity map having an associated dynamic range;

determining a second exposure level for a second image set according to the exposure selection algorithm;

generating the second image set at the determined second exposure level;

generating a second stereo disparity map from the second image set, the second stereo disparity map having an associated dynamic range; and

merging the first stereo disparity map and the second stereo disparity map such that the intensity value for each pixel within a composite stereo disparity map is a linear combination of respective intensity values for corresponding pixels in the first stereo disparity map and the second stereo disparitv map such that the composite stereo disparity map has an effective dynamic range greater than either of the associated dynamic ranges of the first disparity map and the second disparity map.

8. The method of claim 7, wherein the step of determining the second exposure level includes determining an area within the first stereo disparity map comprised by saturated regions and subtracting a predetermined value from the first exposure level if the area exceeds a threshold value.

9. The method of claim 7, wherein the step of determining the second exposure level includes determining an area within the first stereo disparity map comprised by dark regions and adding a predetermined value from the first exposure level if the area exceeds a threshold value.

10. The method of claim 7, wherein the step of determining a first exposure level comprises:

evaluating a first image from a previous iteration to determine the number of saturated pixels within the first image from the previous iteration;

determining a first difference between the determined number of saturated pixels and a target number of pixels; and

calculating the first exposure level as a function of an exposure level associated with the first image from the previous iteration and the determined first difference.

11. The method of claim 10, wherein calculating the first exposure level includes multiplying the determined difference by a predetermined constant to obtain a product and adding the product to the exposure level associated with the first image.

12. The method of claim 10, wherein the step of determining a second exposure level comprises:

evaluating a second image from a previous iteration to determine the number of saturated pixels within the second image from the previous iteration;

determining a second difference between the determined number of saturated pixels within the second image and a target number of pixels; and

calculating the second exposure level as a function of an exposure level associated with the second image from the previous iteration and the determined second difference.

13. The method of claim 12, further comprising:

determining if a first coverage range associated with the first exposure level overlaps a second coverage range associated with the second exposure level; and

determining a third exposure level for a third image set according to an exposure selection algorithm if the first and second coverage ranges do not overlap.

14. A method for generating a stereo disparity map having an enhanced dynamic range, comprising:

determining a desired exposure level for each of a plurality of images, the plurality of images including a first set of images and a second set of images, wherein determining a desired exposure level comprises;

determining the number of pixels falling within each of a plurality of intensity ranges in a region of interest within a first image;

comparing the determined number of pixels for each range to a series of target values for the range; and

selecting an exposure value for a second image as to minimize the sum of the squared differences between the target values and expected values associated with at least one available exposure level;

generating each of the plurality of images at the desired level of exposure;

merging the first set of images to produce a first composite image;

merging the second set of images to produce a second composite image; and

generating a stereo disparity map from the first composite image and the second composite image.

15. The method of claim 14, further comprising evaluating at least one range value associated with the stereo disparity map and discarding at least a portion of the stereo disparity map if one of the at least one range value falls outside of an acceptable range.

16. The method of claim 14, wherein the step of determining a desired level of exposure includes determining an area within a first image from a set of images comprised by dark regions and raising the exposure level for a second image if the area exceeds a threshold value.

17. The method of claim 14, wherein the step of determining a desired level of exposure includes determining an area within a first image from a set of images comprised by saturated regions and lowering the exposure level for a second image if the area exceeds a threshold value.

18. The method of claim 14, wherein the step of determining a desired level of exposure for the plurality of images includes calculating an exposure level for a second image as a function of the difference between an average saturation level over an evaluated region of a first image and a desired average saturation level for the region.