Video manipulation of red, green, blue, distance (RGB-Z) data including segmentation, up-sampling, and background substitution techniques

• US 20110285910A1
• Filed: 12/20/2007
• Published: 11/24/2011
• Est. Priority Date: 06/01/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Crabb, Ryan E., Bamji, Cyrus, Rafii, Abbas
• Granted Patent: US 8,139,142 B2
• Time in Patent Office: Days
• US Class Current: 348/631
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/705 Pixels for depth measurement, e.g. RGBZ

Abstract

RGB-Z imaging systems acquire RGB data typically with a high X-Y resolution RGB pixel array, and acquire Z-depth data with an array of physically larger Z pixels having additive signal properties. In each acquired frame, RGB pixels are mapped to a corresponding Z pixel. Z image resolution is enhanced by identifying Z discontinuities and identifying corresponding RGB pixels where the Z discontinuities occur. Thus segmented data enables RGB background substitution, which preferably blends foreground pixel color and substitute background color. The segmented data also enables up-sampling in which a higher XY resolution Z image with accurate Z values is obtained. Up-sampling uses an equation set enabling assignment of accurate Z values to RGB pixels. Fixed acquisition frame rates are enabled by carefully culling bad Z data. Segmenting and up-sampling enhanced video effects and enable low cost, low Z resolution arrays to function comparably to higher quality, higher resolution Z arrays.

Claims

1. For use with an RGB-Z imaging system useable to image a scene, which RGB-Z imaging system has an RGB array of RGB pixels to acquire RGB data useable to form an RGB image, and has a Z array of Z-pixels having additive signal properties to acquire Z data useable to form a Z-image, a method to increase resolution of said Z image, the method including the following steps:

(a) acquiring said RGB data from said RGB array and acquiring said Z data from said Z array such that at least one RGB pixel in said RGB array corresponds to and is associated with at least one Z pixel in said Z array;

(b) determining from Z data acquired at step (a) where Z discontinuities occur, and for determined said Z discontinuities, identifying from RGB data acquired at step (a) corresponding RGB pixels; and

(c) applying determinations from step (b) to at least approximate a solution to a set of constraints and to at least approximately minimize a set of costs to determine a correct Z value for at said at least one said RGB pixel corresponding to said Z pixel;

wherein a resultant Z-image exhibits enhanced resolution compared to native Z resolution of a Z image produced from said Z array.

2. The method of claim 1, wherein at step (c), said set of constraints utilizes said additive signal properties of said Z pixels.

3. The method of claim 1, wherein at step (c), said additive signal properties of said Z pixels are satisfied by providing said Z pixel array with Z pixels for which a Z pixel response is the sum of responses of associated RGB pixels.

4. The method of claim 1, wherein at step (c), approximately minimizing said set of costs renders Z data at each RGB pixel so as to smooth at least a portion of a surface of an object within said scene imaged by said RGB-Z system and displayed on a resultant image.

5. The method of claim 1, wherein step (b) locates said Z discontinuities using at least one of (i) Z-pixel acquired active brightness discontinuity data, (ii) Z-gradient data, and (iii) RGB pixel data.

6. The method of claim 1, wherein:

said RGB-Z system is disposed in a motor vehicle, said scene is external to said motor vehicle, which scene may include an object hazardous to said motor vehicle; and

said method produces an enhanced Z image useable to protect at least one of (i) said motor vehicle from impact with an object in said scene, (ii) an occupant in said motor vehicle, and (iii) an object in said scene from impact with said motor vehicle.

7. For use with an RGB-Z imaging system useable to image a scene, which RGB-Z imaging system has an RGB array of RGB pixels to acquire RGB data useable to form an RGB image, and has a Z array of Z-pixels having additive signal properties to acquire Z data useable to form a Z-image, a method to identify discontinuities in Z in the RGB image including the steps of:

(a) determining where Z discontinuities occur in said Z array;

(b) acquiring said RGB data and said Z data such that at least one RGB pixel corresponds to and is associated with at least one Z pixel in said Z array at which a Z discontinuity occurs;

(c) using said RGB data to determine where Z discontinuities occur in RGB pixels neighboring said at least one RGB pixel;

wherein discontinuities identifiable in a Z image are now identifiable in an RGB image.

8. The method of claim 7, wherein at step (c) discontinuities in Z are identified using vector-disparity between RGB values.

9. The method of claim 8, wherein at step (c), vector-disparity between RGB values is augmented to include a fourth dimension Z depth estimate for said RGB pixels.

10. The method of claim 7, further including using a confidence map for Z depth values to reduce errors in determination of Z discontinuities at step (c).

11. The method of claim 10, wherein data from Z pixels associated with a low confidence is not used.

12. For use with an RGB-Z imaging system useable to image a scene, which RGB-Z imaging system has an RGB array of RGB pixels to acquire RGB data useable to form an RGB image, and has a Z array of Z-pixels having additive signal properties to acquire Z data useable to form a Z-image, a method to replace RGB data in an RGB image formed by said RGB-Z system, the method including the steps of:

(a) determining at least one location whereat Z discontinuity occurs in said RGB array of RGB pixels such that within a neighboring vicinity of said Z discontinuity at least a first RGB pixel is on one side of said Z discontinuity, and at least a second RGB pixel on a second said of said Z discontinuity; and

wherein;

(b) within said neighboring vicinity of said discontinuity, replacing a value of said at least said first RGB pixel;

(c) within said neighboring vicinity of said discontinuity, leaving unchanged a value of at least said the value of at least second RGB pixel;

wherein said Z discontinuity delineates foreground from background within an RGB image formed by said RGB-Z system, wherein one of said foreground and background is substituted at step (b).

13. The method of claim 12, wherein value replacement for said first RGB pixel at step (b) depends at least in part upon a previous value of said first RGB pixel.