METHOD TO EXTEND LASER DEPTH MAP RANGE

• US 20130050426A1
• Filed: 08/30/2011
• Published: 02/28/2013
• Est. Priority Date: 08/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sarmast, Sam M., Doolittle, Donald L.
• Granted Patent: US 9,491,441 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4863 Detector arrays, e.g. charge-transfer gates
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/133 Equalising the characteristics of different image components, e.g. their average brightness or colour balance
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/73 by influencing the exposure time

Abstract

A method for extending the dynamic range of a depth map by deriving depth information from a synthesized image of a plurality of images captured at different light intensity levels and/or captured over different sensor integration times is described. In some embodiments, an initial image of an environment is captured while the environment is illuminated with light of a first light intensity. One or more subsequent images are subsequently captured while the environment is illuminated with light of one or more different light intensities. The one or more different light intensities may be dynamically configured based on a degree of pixel saturation associated with previously captured images. The initial image and the one or more subsequent images may be synthesized into a synthesized image by applying high dynamic range imaging techniques.

Claims

1. A method for extending depth map range, comprising:

projecting a light pattern into a first environment using a first light intensity level;

capturing a first image of the first environment over a first sensor integration time, the first image is associated with the first light intensity level;

determining a degree of pixel saturation associated with the first image;

setting a second light intensity level different from the first light intensity level based on the degree of pixel saturation;

projecting the light pattern into the first environment using the second light intensity level;

capturing a second image of the first environment, the second image is associated with the second light intensity level; and

synthesizing the first image and the second image into a combined image.

2. The method of claim 1, wherein:

the degree of pixel saturation comprises a percentage of pixels associated with the first image that are saturated; and

the setting a second light intensity level includes setting the second light intensity level to a lower intensity level than the first light intensity level if the percentage of pixels associated with the first image that are saturated is above a predetermined percentage.

3. The method of claim 1, wherein:

the setting a second light intensity level includes setting the second light intensity level to a higher intensity level than the first light intensity level if the degree of pixel saturation is not above a threshold.

4. The method of claim 1, wherein:

the synthesizing the first image and the second image into a combined image includes assigning to a pixel location in the combined image corresponding with a saturated pixel location in the first image a scaled pixel value associated with the second image.

5. The method of claim 1, wherein:

the synthesizing the first image and the second image into a combined image includes applying high dynamic range imaging techniques to the first image and the second image.

6. The method of claim 1, wherein:

the projecting a light pattern into a first environment includes projecting the light pattern using a laser;

the projecting a light pattern into a first environment occurs during a first period of time, the first sensor integration time is subsequent to the first period of time; and

the first light intensity level is associated with a particular distance from the laser.

7. The method of claim 1, wherein:

the capturing a second image of the first environment comprises capturing the second image over a second sensor integration time, the second sensor integration time is less than the first sensor integration time.

8. The method of claim 1, wherein:

the determining a degree of pixel saturation includes determining a number of pixels that have been assigned a maximum light intensity setting;

the light pattern includes horizontal and vertical stripes; and

the second light intensity level is less than the first light intensity level.

9. An electronic device for extending depth map range, comprising:

an illumination source, the illumination source projects a light pattern into a first environment using a first light intensity level;

a sensor, the sensor captures a first image of the first environment over a first sensor integration time, the first image is associated with the first light intensity level; and

one or more processors, the one or more processors determine a degree of pixel saturation associated with the first image, the one or more processors determine a second light intensity level based on the degree of pixel saturation, the illumination source projects the light pattern into the first environment using the second light intensity level, the sensor captures a second image of the first environment, the second image is associated with the second light intensity level, the second light intensity level is different from the first light intensity level, the one or more processors generate a combined image from the first image and the second image.

10. The electronic device of claim 9, wherein:

the degree of pixel saturation comprises a percentage of pixels associated with the first image that are saturated; and

the one or more processors set the second light intensity level to a lower intensity level than the first light intensity level if the percentage of pixels associated with the first image that are saturated is above a particular percentage.

11. The electronic device of claim 9, wherein:

the one or more processors set the second light intensity level to a higher intensity level than the first light intensity level if the degree of pixel saturation is not above a threshold.

12. The electronic device of claim 9, wherein:

the one or more processors generate the combined image by assigning to a pixel location in the combined image corresponding with a saturated pixel location in the first image a scaled pixel value associated with the second image.

13. The electronic device of claim 9, wherein:

the one or more processors generate the combined image by applying high dynamic range imaging techniques to the first image and the second image.

14. The electronic device of claim 9, wherein:

the illumination source comprises a laser;

the illumination source projects the light pattern during a first period of time, the first sensor integration time is subsequent to the first period of time; and

the first light intensity level is associated with a particular distance from the laser.

15. The electronic device of claim 9, wherein:

the sensor captures the second image over a second sensor integration time, the second sensor integration time is less than the first sensor integration time.

16. One or more storage devices containing processor readable code for programming one or more processors to perform a method for extending depth map range comprising the steps of:

projecting a light pattern into a first environment using a first light intensity level, the projecting a light pattern into a first environment occurs during a first period of time;

capturing a first image of the first environment over a first sensor integration time, the first image is associated with the first light intensity level, the first sensor integration time is subsequent to the first period of time;

determining a degree of pixel saturation associated with the first image;

setting a second light intensity level based on the degree of pixel saturation;

projecting the light pattern into the first environment using the second light intensity level;

capturing a second image of the first environment, the second image is associated with the second light intensity level, the second light intensity level is different from the first light intensity level;

synthesizing the first image and the second image into a combined image; and

generating a depth map based on the combined image.

17. The one or more storage devices of claim 16, wherein:

the projecting a light pattern into a first environment includes projecting the light pattern using a laser; and

the first light intensity level is associated with a particular distance from the laser.

18. The one or more storage devices of claim 17, wherein:

the degree of pixel saturation comprises a percentage of pixels associated with the first image that are saturated; and

the setting a second light intensity level includes setting the second light intensity level to a lower intensity level than the first light intensity level if the percentage of pixels associated with the first image that are saturated is above a predetermined percentage.

19. The one or more storage devices of claim 17, wherein:

the setting a second light intensity level includes setting the second light intensity level to a higher intensity level than the first light intensity level if the degree of pixel saturation is not above a particular threshold.

20. The one or more storage devices of claim 19, wherein:

the synthesizing the first image and the second image into a combined image includes assigning to a pixel location in the combined image corresponding with a saturated pixel location in the first image a scaled pixel value associated with the second image.