Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing

• US 20060221250A1
• Filed: 06/01/2006
• Published: 10/05/2006
• Est. Priority Date: 01/28/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Zhao, Peiqian, Rossbach, Christopher J., Rafii, Abbas
• Granted Patent: US 8,134,637 B2
• Time in Patent Office: Days
• US Class Current: 348/630
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements

Abstract

An imaging system substantially simultaneously acquires z-depth and brightness data from first sensors, and acquires higher resolution RGB data from second sensors, and fuses data from the first and second sensors to model an RGBZ image whose resolution can be as high as resolution of the second sensors. Time correlation of captured data from first and second sensors is associated with captured image data, which permits arbitrary mapping between the two data sources, ranging from 1:many to many:1. Preferably pixels from each set of sensors that image the same target point are mapped. Many z-depth sensor settings may be used to create a static environmental model. Non-correlative and correlative filtering is carried out, and up-sampling to increase z-resolution occurs, from which a three-dimensional model is constructed using registration and calibration data.

Claims

1. For use with a sensor system that includes first sensors to acquire z-depth and brightness data at a first resolution, and second sensors to acquire RGB data at a higher second resolution, said first and second sensors acquiring data substantially simultaneously, a method to combine data acquired by said first sensors with data acquired by said second sensors such that an RGBZ image may be rendered whose resolution is higher than said first resolution, the method including the following steps:

(a) correlating timing information with data acquired by said first sensors and with data acquired by said second sensors; and

(b) up-sampling and increasing resolution of data acquired by said first sensors such that an RGBZ model may be rendered from data so up-sampled and increased in resolution, resolution of said RGBZ model exceeding said first resolution of said first sensors.

2. The method of claim 1, wherein said first sensors have a characteristic selected from a group consisting of (i) said first sensors are fabricated on a common integrated circuit array as said second sensors, and (ii) said first sensors are fabricated on an integrated circuit array discrete from an integrated circuit array that includes said second sensors.

3. The method of claim 1, further including, intermediate step (a) and step (b), a step of filtering data acquired by at least one of said first sensors and said second sensors.

4. The method of claim 1, where at step (a), a mapping relationship between data acquired by said first sensors and data acquired by said second sensors has a relationship selected from a group consisting of (i) one-to-one, (ii) one-to-many, and (iii) many-to-one.

5. The method of claim 1, wherein step (a) includes calibration and registration of data acquired by said first sensors and data acquired by said second sensors, such that pixels associated with said first sensors that image a point on a target are mapped with pixels associated with said second sensors that image said same point on said target.

6. The method of claim 1, wherein step (a) further includes capturing from said first sensors depth z, brightness, active brightness, and capturing from said second sensors RGB data, substantially simultaneously in real-time.

7. The method of claim 3, wherein said step of filtering data includes at least one of (i) non-correlative filtering, (ii) non-correlative filtering and correction, (iii) correlative filtering, and (iv) correlative filtering and correction;

wherein correlative filtering includes using data from said first sensors to generate a measure of confidence on data from said second sensors, and vice versa.

8. The method of claim 3, where said step of filtering data includes a filtering characteristic selected from a group consisting of (i) filtering data acquired by said first sensors is carried out independently of filtering data acquired by said second sensors, and (ii) filtering data acquired by said first sensors is carried out with reference to filtering data acquired by said second sensors.

9. The method of claim 1, wherein at step (b), said RGBZ model is characterized by at least one of (I) parameters including at least three of (i) actual (x,y,z) spatial dimension, (ii) RGB color values, (iii) brightness, and (iv) active brightness, and (II) said RGBZ model is representable by a fusion image data stream of matrices M_T having R×

C data points, where each matrix element includes at least spatial data in three dimensions (x,y,z), brightness, active brightness, and a confidence metric p that estimates probability that any inferred spatial data are substantially correct.

10. The method of claim 1, wherein step (b) includes at least one of (i) creating a sparse range-z map that is filled-in using data from said second sensors, to make logical inferences upon implied z-measurements, (ii) an interpolation process including edge-based detection, (iii) an interpolation process including gradient-based detection, (iii) an interpolation process including threshold testing, (iv) a Lagrange interpolation, and (v) detection of at least one of RGB image color, texture and shading.

11. A time-of-flight system that acquires z-depth, brightness, and RGB information and outputs data from which an RGBZ image may be rendered, the range-finding system including:

a plurality of first sensors to acquire z-depth and brightness data at a first resolution;

a plurality of second sensors to acquire RGB data at a higher second resolution, said first and second sensors acquiring data substantially simultaneously, said first sensors having a characteristic selected from a group consisting of (i) said first sensors are fabricated on a common integrated circuit array as said second sensors, and (ii) said first sensors are fabricated on an integrated circuit array discrete from an integrated circuit array that includes said second sensors; and

further including

means for combining data acquired by said first sensors with data acquired by said second sensors such that said RGBZ image may be rendered whose resolution is higher than said first resolution.

12. The system of claim 11, further including means for filtering data acquired by at least one of said first sensors and said second sensors.

13. The system of claim 11, wherein said means for combining includes:

means for correlating timing information with data acquired by said first sensors and with data acquired by said second sensors; and

means for up-sampling and increasing resolution of data acquired by said first sensors to a resolution not required to be bounded by said second resolution, such that an RGBZ model may be rendered from data so up-sampled and increased in resolution.

14. The system of claim 11, wherein said means for combining includes a software routine that upon execution by a processor carries out said combining.

15. The system of claim 11, wherein said means for correlating timing information functions with a mapping relationship between data acquired by said first sensors and data acquired by said second sensors, said mapping relationship having a characteristic selected from a group consisting of (i) one-to-one, (ii) one-to-many, and (iii) many-to-one.

16. The system of claim 11, wherein said means for correlating timing information has at least one characteristic selected from a group consisting of (i) said means for correlating timing information carries out calibration and registration of data acquired by said first sensors and data acquired by said second sensors such that pixels associated with said first sensors that image a point on a target are mapped with pixels associated with said second sensors that image said same point on said target, and (ii) said means for correlating timing information captures from said first sensors depth z, brightness, active brightness, and captures from said second sensors RGB data, substantially simultaneously in real-time.

17. The system of claim 12, wherein said means for filtering data includes at least one of (i) non-correlative filtering, (ii) non-correlative filter and correction, (iii) correlative filtering, and (iv) correlative filtering and correction;

wherein correlative filtering includes using data from said first sensors to generate a measure of confidence on data from said second sensors, and vice versa.

18. The system of claim 13, where said means for filtering data has a characteristic selected from a group consisting of selected from a group consisting of (i) filtering data acquired by said first sensors is carried out independently of filtering data acquired by said second sensors, and (ii) filtering data acquired by said first sensors is carried out with reference to filtering data acquired by said second sensors.

19. The system of claim 11, wherein said RGBZ model obtainable from said means for up-sampling and increasing resolution is characterized by at least one of (I) parameters including at least three of (i) actual (x,y,z) spatial dimension, (ii) RGB color values, (iii) brightness, and (iv) active brightness, and (II) said RGBZ model is representable by a fusion image data stream of matrices MT having R×

C data points, where each matrix element includes at least spatial data in three dimensions (x,y,z), brightness, active brightness, and a confidence metric p that estimates probability that any inferred spatial data are substantially correct.

20. The system of claim 13, wherein said means for up-sampling and increasing resolution has at least one characteristic selected from a group consisting of (i) said means for up-sampling and increasing resolution creates a sparse range-z map that is filled-in using data from said second sensors, to make logical inferences upon implied z-measurements, and (II) said means for up-sampling and increasing resolution implements at least one type of computation selected from a group consisting of (i) edge-based detection, (ii) gradient-based detection, (iii) threshold testing, (iv) Lagrange interpolation, and (v) detection of at least one of RGB image color, texture, and shading.