Method and system to increase dynamic range of time-of-flight (TOF) and/or imaging sensors

• US 7,379,100 B2
• Filed: 02/14/2005
• Issued: 05/27/2008
• Est. Priority Date: 02/12/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Gokturk, Salih Burak, Rafii, Abbas
• Primary Examiner(s): Hannett; James M.
• Application Number:
  US11/058,028
• Publication Number:
  US 20060114333A1
• Time in Patent Office: 1,198 Days
• Field of Search: 348/229.1
• US Class Current: 348/229.1
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4865 Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak peak detection in noise, signal conditioning G01S7/487
• CPC Class Codes: G01S 7/4868 Controlling received signal intensity or exposure of sensor
• CPC Class Codes: G01S 7/487 Extracting wanted echo signals , e.g. pulse detection
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Dynamic range of photodetector sensors useable in a TOF system is enhanced by capturing images of an object using multiple exposure time settings. Longer exposure settings more appropriately capture non-reflective and/or distant objects, while shorter exposure settings more appropriately capture reflective and/or closer objects. During parallel mode operation, detection signal readouts are taken from each photodetector at different time intervals within an overall exposure time. In sequential mode operation, detection signal readouts are taken and stored for each photodetector at the end of a first exposure time interval and the photodetectors are reset. After a second, different exposure time interval readouts are taken and stored, and the photodetectors reset, etc. In these modes one of the time exposure intervals will be relatively optimum for enhanced dynamic range operation. Once images with multiple exposure settings are obtained, best effort brightness and range images can be obtained, and motion artifacts can be reduced.

Claims

1. A method to enhance dynamic range of an array of photodetectors responsive to incoming optical energy returned from an object a distance Z away to determine at least distance Z to said object, said photodetectors having a dynamic range in dB of DR before application of the method, the method including:

(a) enabling said photodetectors to receive said incoming optical energy during a long time interval;

(b) within said long time interval, at the end of each of a number n of unequal time duration sub-intervals storing a detection output signal from each of said photodetectors, where n>

1 and wherein at least a pair of said unequal time duration sub-intervals, denoted shutters and shutter_i-1, satisfy a relationship

2. The method of claim 1, wherein at step (b) said unequal time duration sub-intervals are related to each other in a manner selected from a group consisting of (i) linearly, (ii) non-linearly, and (iii) logarithmically.

3. The method of claim 1, wherein at step (c), determining is based upon examination of images formed from at least one of (i) luminosity components of said incoming optical energy detected by said photodetectors, (ii) variation in ranges of Z in range images, and (iii) magnitude of signal-to-noise in ranges of Z in range images, and further including assigning a range associated with a best quality detection signal as a final measure of range Z.

4. The method of claim 1 wherein if n=2, improvement G in dB of dynamic range of said photodetectors is given by

5. The method of claim 1, wherein step (d) includes selectively repeating step (b) for at least some of said photodetectors determined at step (c) to be functioning marginally within their operational dynamic range.

6. The method of claim 1, further including adjusting bias potential to at least some photodetectors determined at step (c) to be functioning marginally within their operational dynamic range.

7. The method of claim 1, wherein step (c) includes at least one step selected from a group consisting of:

(i) examination of images formed from luminosity components of said incoming optical energy detected by said photodetectors;

(ii) mapping all said images to a common brightness level; and

(iii) increasing dynamic range of said photodetectors by applying one of (1) non-linear transformation, (2) histographic equalization of normalized common brightness levels, (3) a linear stretching function to said common brightness level; and

wherein step (c) also includes setting all brightness signals to a common brightness level, and using a most recently determined said common brightness level as a final measure of brightness.

8. The method of claim 1, wherein each step is carried out under microprocessor control using a microprocessor and memory fabricated on a common integrated circuit substrate containing at least said array of photodetectors.

9. A method to enhance dynamic range of an array of photodetectors responsive to incoming optical energy returned from an object a distance Z away to determine at least distance Z to said object, said photodetectors having a dynamic range in dB of DR before application of the method, the method including:

(a) controllably enabling said photodetectors to receive said incoming optical energy during n sequential unequal time duration sub-intervals;

(b) at the end of each of said n unequal time duration sub-intervals storing a detection output signal from each of said photodetectors, where n>

1 and wherein at least a pair of said unequal time duration sub-intervals, denoted shutter_i and shutter_i-1, satisfy a relationship

10. The method of claim 9, wherein at step (b) said unequal time duration sub-intervals are related to each other in a manner selected from a group consisting of (i) linearly, (ii) non-linearly, and (iii) logarithmically.

11. The method of claim 9, wherein at step (c), determining is based upon examination of images formed from at least one of (i) luminosity components of said incoming optical energy detected by said photodetectors, (ii) variation in ranges of Z in range images, and (iii) magnitude of signal-to-noise in ranges of Z in range images, and further including assigning a range associated with a best quality detection signal as a final measure of range Z.

12. The method of claim 9, wherein if n=2, improvement G in dB of dynamic range in dB of said photodetectors is given by

13. The method of claim 9, wherein step (d) includes selectively repeating step (b) for at least some of said photodetectors determined at step (c) to be functioning marginally within their operational dynamic range.

14. The method of claim 9, further including adjusting bias potential to at least some photodetectors determined at step (c) to be functioning marginally within their operational dynamic range.

15. The method of claim 9, wherein step (c) includes at least one step selected from a group consisting of:

(I) examination of images formed from luminosity components of said incoming optical energy detected by said photodetectors;

(ii) mapping all said images to a common brightness level; and

(iii) increasing dynamic range of said photodetectors by applying one of (1) non-linear transformation, (2) histographic equalization of normalized common brightness levels, and (3) a linear stretching function to said common brightness level; and

wherein step (c) also after setting all brightness signals to a common brightness level, and assigning said common level of brightness as a final measure of brightness;

wherein step (c) also after setting all brightness signals to a common brightness level, and assigning said common level of brightness as a final measure of brightness; and

wherein step (c) also includes setting all brightness signals to a common brightness level, and using a most recently determined said common brightness level as a final measure of brightness.

16. The method of claim 9, wherein each step is carried out under microprocessor control using a microprocessor and memory fabricated on a common integrated circuit substrate containing at least said array of photodetectors.

17. A CMOS-implementable system to enhance dynamic range of an array of photodetectors responsive to incoming optical energy returned from an object a distance Z away to determine at least distance Z to said object, said photodetectors having a dynamic range in dB of DR before enhancement of dynamic range, the system including:

means for enabling said photodetectors to receive said incoming optical energy during a long time interval;

means for storing, within said long time interval at the end of each of a number n of unequal time duration sub-intervals, a detection output signal from each of said photodetectors, where n>

1, wherein at least a pair of said unequal time duration sub-intervals, denoted shutter_i and shutter_i-1, satisfy a relationship

18. The system of claim 17, wherein if n=2, improvement G in dB of dynamic range of said photodetectors is given by

19. The system of claim 17, wherein said means for determining includes examination of images formed at least one of (i) luminosity components of said incoming optical energy detected by said photodetectors, (ii) variation in ranges of Z in range images, and (iii) magnitude of signal-to-noise in ranges of Z in range images.

20. A CMOS-implementable system to enhance dynamic range of an array of photodetectors responsive to incoming optical energy returned from an object a distance Z away to determine at least distance Z to said object, said photodetectors having a dynamic range in dB of DR before enhancement of dynamic range, the system including:

means for controllably enabling said photodetectors to receive said incoming optical energy during n sequential unequal time duration sub-intervals, wherein adjacent said unequal time duration sub-intervals are separated from each other in a manner selected from a group consisting of (i) linearly, (ii) non-linearly, and (iii) logarithmically;

means for storing, at the end of each of said n unequal time duration sub-intervals, a detection output signal from each of said photodetectors, where n>

1 and wherein at least a pair of said unequal time duration sub-intervals, denoted shutter_i and shutter_i-1, satisfy relationship

21. The system of claim 20, wherein if n=2, improvement G in dB of dynamic range of said photodetectors is given by

22. The system of claim 20, wherein said means for determining includes examination of images formed at least one of (i) luminosity components of said incoming optical energy detected by said photodetectors, (ii) variation in ranges of Z in range images, and (iii) magnitude of signal-to-noise in ranges of Z in range images.