Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing

• US 8,134,637 B2
• Filed: 06/01/2006
• Issued: 03/13/2012
• Est. Priority Date: 01/28/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rossbach, Christopher J., Rafii, Abbas, Zhao, Peiqian
• Primary Examiner(s): Ometz, David
• Assistant Examiner(s): DURNFORD GESZVAIN, DILLON
• Application Number:
  US11/444,947
• Publication Number:
  US 20060221250A1
• Time in Patent Office: 2,112 Days
• Field of Search: 348/274, 348/275, 348/336, 348/337, 348/348, 348/349, 348/350, 348 42- 60, 356 501- 508
• US Class Current: 348/348
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements

Abstract

An imaging system substantially simultaneously acquires z-depth and brightness data from first sensors, and acquires higher resolution RGB data from second sensors, and fuses data from the first and second sensors to model an RGBZ image whose resolution can be as high as resolution of the second sensors. Time correlation of captured data from first and second sensors is associated with captured image data, which permits arbitrary mapping between the two data sources, ranging from 1:many to many:1. Preferably pixels from each set of sensors that image the same target point are mapped. Many z-depth sensor settings may be used to create a static environmental model. Non-correlative and correlative filtering is carried out, and up-sampling to increase z-resolution occurs, from which a three-dimensional model is constructed using registration and calibration data.

Claims

1. For use with a sensor system that includes first sensors to acquire from a scene z-depth values and brightness data values at a first resolution, each of said first sensors occupying a first sensor area, and second sensors to acquire from said scene RGB data values at a higher second resolution, each of said second sensors occupying a second sensor area, said first and second sensors acquiring data values substantially simultaneously, a method to combine data values acquired by said first sensors with data values acquired by said second sensors such that an RGBZ image may be rendered whose resolution is higher than said first resolution, the method including the following steps:

(a) dynamically correlating data values acquired by said first sensors and data values acquired by said second sensors such that depth data computed for a higher resolution pixel are affected by data values from said second sensors; and

(b) up-sampling and increasing resolution of data values acquired by said first sensors such that an RGBZ model may be rendered from data values so up-sampled and increased in resolution, resolution of said RGBZ model exceeding said first resolution of said first sensors.

2. The method of claim 1, wherein said first sensors have a characteristic selected from a group consisting of (i) said first sensors are fabricated on a common integrated circuit array as said second sensors, (ii) said first sensors are fabricated on an integrated circuit array discrete from an integrated circuit array that includes said second sensors, and (iii) said first sensor area is greater than said second sensor area.

3. The method of claim 1, further including, intermediate step (a) and step (b), a step of filtering data values acquired by at least one of said first sensors and said second sensors.

4. The method of claim 1, where step (a) includes mapping a relationship between data values acquired by said first sensors and data values acquired by said second sensors, said mapping having a relationship selected from a group consisting of (i) one-to-one, (ii) one-to-many, and (iii) many-to-one.

5. The method of claim 1, wherein step (a) includes calibration and registration of data values acquired by said first sensors and data values acquired by said second sensors, such that pixels associated with said first sensors that image a point on a target are mapped with pixels associated with said second sensors that image said same point on said target.

6. The method of claim 1, wherein step (a) further includes capturing from said first sensors depth z values, brightness, active brightness values, and capturing from said second sensors RGB data values, substantially simultaneously in real-time.

7. The method of claim 3, wherein said step of filtering data values includes at least one of (i) non-correlative filtering, (ii) non-correlative filtering and correction, (iii) correlative filtering, and (iv) correlative filtering and correction;

wherein correlative filtering includes using data values from said first sensors to generate a measure of confidence on data values from said second sensors, and vice versa.

8. The method of claim 3, where said step of filtering data values includes a filtering characteristic selected from a group consisting of (i) filtering data values acquired by said first sensors is carried out independently of filtering data values acquired by said second sensors, and (ii) filtering data values acquired by said first sensors is carried out with reference to filtering data values acquired by said second sensors.

9. The method of claim 1, wherein at step (b), said RGBZ model is characterized by at least one of (I) parameters including at least three of (i) actual (x,y,z) spatial dimension values, (ii) RGB color values, (iii) brightness values, and (iv) active brightness values, and (II) said RGBZ model is representable by a fusion image data stream of matrices M_T having R×

C data value points, where each matrix element includes at least spatial data values in three dimensions (x,y,z), brightness values, active brightness values, and a confidence metric p that estimates probability that any inferred spatial data values are substantially correct.

10. The method of claim 1, wherein step (b) includes at least one of (i) creating a sparse range-z map that is filled-in using data values from said second sensors, to make logical inferences upon implied z-measurements, (ii) an interpolation process including edge-based detection, (iii) an interpolation process including gradient-based detection, (iv) an interpolation process including threshold testing, and (v) a Lagrange interpolation.

11. A time-of-flight system that acquires z-depth values, brightness values, and RGB information values and outputs data values from which an RGBZ image may be rendered, the time-of-flight system including:

a plurality of first sensors to acquire from a scene z-depth values and brightness data values at a first resolution, each of said first sensors occupying a first sensor area;

a plurality of second sensors to acquire from said scene RGB data values at a higher second resolution, each of said second sensors occupying a second sensor area, said first and second sensors acquiring data values substantially simultaneously, said first sensors having a characteristic selected from a group consisting of (i) said first sensors are fabricated on a common integrated circuit array as said second sensors, (ii) said first sensors are fabricated on an integrated circuit array discrete from an integrated circuit array that includes said second sensors, and (iii) said first sensor area is greater than said second sensor area, and further including;

means for dynamically correlating data values acquired by said first sensors with data values acquired by said second sensors such that said RGBZ image may be rendered whose resolution is higher than said first resolution, in which RGB image data values from said second sensors affect depth data of a higher resolution pixel that is computed based on data values of said first sensor.

12. The system of claim 11, further including means for filtering data values acquired by at least one of said first sensors and said second sensors.

13. The system of claim 11, wherein said means for dynamically correlating includes:

means for correlating timing information with data values acquired by said first sensors and with data values acquired by said second sensors; and

means for up-sampling and increasing resolution of data values acquired by said first sensors to a resolution not required to be bounded by said second resolution, such that an RGBZ model may be rendered from data values so up-sampled and increased in resolution.

14. The system of claim 11, wherein said means for dynamically correlating includes a software routine that upon execution by a processor carries out said combining.

15. The system of claim 13, wherein said means for correlating timing information functions with a mapping relationship between data values acquired by said first sensors and data values acquired by said second sensors, said mapping relationship having a characteristic selected from a group consisting of (i) one-to-one, (ii) one-to-many, and (iii) many-to-one.

16. The system of claim 13, wherein said means for correlating timing information has at least one characteristic selected from a group consisting of (i) said means for correlating timing information carries out calibration and registration of data values acquired by said first sensors and data values acquired by said second sensors such that pixels associated with said first sensors that image a point on a target are mapped with pixels associated with said second sensors that image said same point on said target, and (ii) said means for correlating timing information captures from said first sensors depth z values, brightness values, active brightness values, and captures from said second sensors RGB data values, substantially simultaneously in real-time.

17. The system of claim 12, wherein said means for filtering data values includes at least one of (i) non-correlative filtering, (ii) non-correlative filter and correction, (iii) correlative filtering, and (iv) correlative filtering and correction;

wherein correlative filtering includes using data values from said first sensors to generate a measure of confidence on data values from said second sensors, and vice versa.

18. The system of claim 12, where said means for filtering data values has a characteristic selected from a group consisting of selected from a group consisting of (i) filtering data values acquired by said first sensors is carried out independently of filtering data values acquired by said second sensors, and (ii) filtering data values acquired by said first sensors is carried out with reference to filtering data values acquired by said second sensors.

19. The system of claim 13, wherein said RGBZ model obtainable from said means for up-sampling and increasing resolution is characterized by at least one of (I) parameters including at least three of (i) actual (x,y,z) spatial dimension values, (ii) RGB color values, (iii) brightness values, and (iv) active brightness values, and (II) said RGBZ model is representable by a fusion image data stream of matrices M_T having R×

C data value points, where each matrix element includes at least spatial data in three dimensions (x,y,z), brightness values, active brightness values, and a confidence metric p that estimates probability that any inferred spatial data values are substantially correct.

20. The system of claim 13, wherein said means for up-sampling and increasing resolution has at least one characteristic selected from a group consisting of (I) said means for up-sampling and increasing resolution creates a sparse range-z map that is filled-in using data values from said second sensors, to make logical inferences upon implied z-measurements, and (II) said means for up-sampling and increasing resolution implements at least one type of interpolation selected from a group consisting of (i) edge-based detection, (ii) gradient-based detection, (iii) threshold testing, and (iv) Lagrange interpolation.