Time of Flight Sensor Binning

• US 20140253688A1
• Filed: 03/11/2013
• Published: 09/11/2014
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Metz, Werner Adam, Ko, Dong-IK
• Granted Patent: US 9,134,114 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G01B 11/026 by measuring distance between sensor and object G01B11/0608 takes precedence
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/48 of systems according to group G01S17/00
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4865 Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak peak detection in noise, signal conditioning G01S7/487
• CPC Class Codes: G01S 7/4911 Transmitters
• CPC Class Codes: G01S 7/4915 Time delay measurement, e.g. operational details for pixel components signal extraction and conditioning G01S7/493; Phase measurement
• CPC Class Codes: G01S 7/4918 Controlling received signal intensity, gain or exposure of sensor

Abstract

A time-of-flight sensor device generates and analyzes a high-resolution depth map frame from a high-resolution image to determine a mode of operation for the time-of-flight sensor and an illuminator and to control the time-of-flight sensor and illuminator according to the mode of operation. A binned depth map frame can be created from a binned image from the time-of-flight sensor and combined with the high-resolution depth map frame to create a compensated depth map frame.

Claims

1. A method for use with a time-of-flight sensor, the method comprising:

generating by at least one processing device at least one high-resolution depth map frame from at least one high-resolution image received from the time-of-flight sensor;

analyzing by the at least one processing device at least one of the at least one high-resolution depth map frame and the at least one high-resolution image;

determining by the at least one processing device a mode of operation having an illuminator power output level and a binning factor for at least one binned lower-resolution image;

generating at the at least one processing device at least one binned lower-resolution depth map frame from the at least one binned lower-resolution image received from the time-of-flight sensor;

combining information from the at least one high-resolution depth map frame with information from the at least one binned lower-resolution depth map frame to generate at least one compensated depth map frame; and

controlling by the at least one processing device the time-of-flight sensor and at least one illuminator according to the mode of operation.

2. The method of claim 1, wherein the step of analyzing by the at least one processing device the at least one of the at least one high-resolution depth map frame and the at least one high-resolution image further comprises determining an ambient light interference level, wherein determining by the at least one processing device a mode of operation comprising an illuminator power output level further comprises:

determining whether the ambient light interference level is lower than a threshold ambient light interference level; and

determining a reduced illuminator power output level in response to determining that the ambient light interference level is lower than a threshold ambient light interference level;

and wherein the controlling by the at least one processing device the at least one illuminator according to the mode of operation further comprises reducing illuminator power output level according to the determined reduced illuminator power output level.

3. The method of claim 1, wherein:

analyzing by the at least one processing device the at least one of the at least one high-resolution depth map frame and the at least one high-resolution image further comprises determining an ambient light interference level;

wherein determining by the at least one processing device a mode of operation comprising the binning factor for the at least one binned lower-resolution image further comprises;

determining whether the ambient light interference level is higher than a threshold ambient light interference level; and

determining an increased binning factor for the at least one binned lower-resolution image in response to determining that the ambient light interference level is higher than a threshold ambient light interference level;

and wherein the controlling by the at least one processing device the time-of-flight sensor according to the mode of operation further comprises capturing the at least one binned lower-resolution image according to the increased binning factor.

4. The method of claim 1, wherein:

analyzing by the at least one processing device the at least one of the at least one high-resolution depth map frame and the at least one high-resolution image further comprises determining a depth of at least one foreground object;

determining by the at least one processing device a mode of operation comprising an illuminator power output level further comprises;

determining whether the depth of the at least one foreground object is lower than a threshold depth; and

determining a reduced illuminator power output level in response to determining that the depth of the at least one foreground object is lower than a threshold depth;

and wherein the controlling by the at least one processing device the at least one illuminator according to the mode of operation further comprises reducing illuminator power output level according to the determined reduced illuminator power output level.

5. The method of claim 1, wherein:

analyzing by the at least one processing device the at least one of the at least one high-resolution depth map frame and the at least one high-resolution image further comprises determining a depth of at least one foreground object;

determining by the at least one processing device a mode of operation comprising the binning factor for the at least one binned lower-resolution image further comprises;

determining whether the depth of the at least one foreground object is lower than a threshold depth; and

determining an increased binning factor for the at least one binned lower-resolution image in response to determining that the depth of the at least one foreground object is lower than a threshold depth;

and wherein the controlling by the at least one processing device the time-of-flight sensor according to the mode of operation further comprises capturing the at least one binned lower-resolution image according to the increased binning factor.

6. The method of claim 1, wherein the at least one high-resolution image comprises at least one binned image with a binning factor less-than or equal to the binning factor for the at least one binned lower-resolution image.

7. The method of claim 1, wherein analyzing the at least one of the at least one high-resolution depth map frame and the at least one high-resolution image further comprises analyzing the at least one of the at least one high-resolution depth map frame and the at least one high-resolution image in combination with at least one of at least one previous image and at least one previous depth map frame.

8. An apparatus comprising:

at least one illuminator;

at least one time-of-flight image sensor;

at least one processing device operatively coupled to the at least one illuminator and the at least one time-of-flight image sensor, the at least one processing device configured to;

receive at least one high-resolution image from the at least one time-of-flight image sensor;

generate at least one high-resolution depth map frame using the at least one high-resolution image;

analyze using a scene analysis module at least one of the at least one high-resolution image and the at least one high-resolution depth map frame;

determine using a binning mode configuration module and the analysis from the scene analysis module a mode of operation comprising an illuminator power output level and a binning factor for at least one binned lower-resolution image;

receive the at least one binned lower-resolution image from the at least one time-of-flight image sensor;

generate at least one binned lower-resolution depth map frame using the at least one binned lower-resolution image;

combine information from the at least one high-resolution depth map frame with information from the at least one binned lower-resolution depth map frame to generate at least one compensated depth map frame; and

control operation of the at least one time-of-flight image sensor and the at least one illuminator according to the mode of operation.

9. The apparatus of claim 8, wherein the at least one processing device is further configured to:

analyze using the scene analysis module the at least one of the at least high-resolution depth map frame and the at least one high-resolution image to determine an ambient light interference level;

determine a mode of operation by;

determining whether the ambient light interference level is lower than a threshold ambient light interference level; and

determining a reduced illuminator power output level in response to determining that the ambient light interference level is lower than a threshold ambient light interference level;

and

control operation of the at least one illuminator according to the determined reduced illuminator power output level.

10. The apparatus of claim 8, wherein the at least one processing device is further configured to:

analyze using the scene analysis module the at least one of the at least high-resolution depth map frame and the at least one high-resolution image to determine an ambient light interference level;

determine the binning factor for the at least one binned lower-resolution image by;

determining whether the ambient light interference level is higher than a threshold ambient light interference level; and

determining an increased binning factor for the at least one binned lower-resolution image in response to determining that the ambient light interference level is higher than a threshold ambient light interference level;

and

capture the at least one binned lower-resolution image according to the increased binning factor.

11. The apparatus of claim 8, wherein the at least one processing device is further configured to:

analyze using the scene analysis module the at least one high-resolution depth map frame to determine a depth of at least one foreground object;

determine whether the depth of the at least one foreground object is lower than a threshold depth;

determine a reduced illuminator power output level in response to determining that the depth of the at least one foreground object is lower than a threshold depth; and

control operation of the at least one illuminator by reducing illuminator power output level according to the determined reduced illuminator power output level.

12. The apparatus of claim 8, wherein the at least one processing device is further configured to:

analyze using the scene analysis module the at least one high-resolution depth map frame to determine a depth of at least one foreground object;

determine whether the depth of the at least one foreground object is lower than a threshold depth;

determine an increased binning factor for the at least one binned lower-resolution image in response to determining that the depth of the at least one foreground object is lower than a threshold depth; and

control operation of the time-of-flight sensor according to the mode of operation by capturing the at least one binned lower-resolution image according to the increased binning factor.

13. The apparatus of claim 8, wherein the at least one time-of-flight sensor further comprises:

a plurality of sensor pixels, wherein the time-of-flight sensor is configured to temporarily store individual information from at least one sensor pixel of the plurality of sensor pixels and to temporarily store information from the at least one sensor pixel aggregated with information from at least one other sensor pixel of the plurality of sensor pixels.

14. The apparatus of claim 13, wherein the time-of-flight sensor is configured to operate in a full-resolution mode and a binned mode, and wherein the time-of-flight sensor is further configured to temporarily store individual information from the at least one sensor pixel when operating in the full-resolution mode and to temporarily store information from the at least one sensor pixel aggregated with information from the at least one other sensor pixels when operating in the binned mode.

15. The apparatus of claim 8, wherein the at least one high-resolution image comprises at least one binned image with a binning factor less-than or equal to the binning factor for the at least one binned lower-resolution image.

16. The apparatus of claim 8, wherein the at least one processing device is further configured to analyze the at least one of the at least one high-resolution image and the at least one high-resolution depth map frame by analyzing the at least one of the at least one high-resolution image and the at least one high-resolution depth map frame in combination with at least one of a previous image and a previous depth map frame.

17. An apparatus comprising:

at least one infrared illuminator;

at least one time-of-flight image sensor having a plurality of analog pixel signal aggregators configured to aggregate a plurality of analog pixel signals to a plurality of binned analog pixel signals to form at least one binned image;

at least one processing device operatively coupled to the at least one infrared illuminator and the at least one time-of-flight image sensor, the at least one processing device configured to;

receive at least one high-resolution image from the at least one time-of-flight image sensor;

generate at least one high-resolution depth map frame using the at least one high-resolution image;

analyze at least one of the at least one high-resolution image or the at least one high-resolution depth map frame using an ambient light analysis module to determine an ambient light interference level;

analyze the at least one high-resolution depth map frame using a scene analysis module to determine a depth of at least one foreground object;

determine using a binning mode configuration module, using the ambient light interference level, and using the depth of the at least one foreground object, a mode of operation having a calculated infrared illuminator power output level and a binning factor for at least one binned lower-resolution image;

adjust the infrared illuminator power output level according to the calculated infrared illuminator power output level;

receive at least one binned lower-resolution image from the at least one time-of-flight image sensor according to the binning factor;

generate at least one binned depth map frame using the at least one binned image; and

combine information from the at least one high-resolution depth map frame with information from the at least one binned depth map frame to generate at least one compensated frame.

18. The apparatus of claim 17, wherein the at least one time-of-flight image sensor is configured to operate in a full-resolution mode and at least one binned mode and to produce a full-resolution image comprising information relating to individual ones of the plurality of analog pixel signals when operating in the full-resolution mode and to produce a binned image comprising information relating to individual ones of the plurality of binned analog pixel signals when operating in the at least one binned mode.