DE-ALIASING DEPTH IMAGES

• US 20110234756A1
• Filed: 03/26/2010
• Published: 09/29/2011
• Est. Priority Date: 03/26/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gilboa, Guy, Katz, Sagi, Tardif, John, Adler, Avishai
• Granted Patent: US 8,514,269 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

Techniques are provided for de-aliasing depth images. The depth image may have been generated based on phase differences between a transmitted and received modulated light beam. A method may include accessing a depth image that has a depth value for a plurality of locations in the depth image. Each location has one or more neighbor locations. Potential depth values are determined for each of the plurality of locations based on the depth value in the depth image for the location and potential aliasing in the depth image. A cost function is determined based on differences between the potential depth values of each location and its neighboring locations. Determining the cost function includes assigning a higher cost for greater differences in potential depth values between neighboring locations. The cost function is substantially minimized to select one of the potential depth values for each of the locations.

Claims

1. A machine-implemented method, comprising:

accessing a depth image that includes a depth value for each of a plurality of locations in the depth image, each of the locations has one or more neighbor locations;

determining potential depth values for each of the plurality of locations based on the depth value in the depth image for the location and potential aliasing in the depth image;

determining a cost function based on differences between the potential depth values of each location and its one or more neighbor locations, determining the cost function includes assigning a higher cost for greater differences in potential depth values between a pair of neighbor locations; and

substantially minimizing the cost function to select one of the potential depth values for each of the locations.

2. The machine-implemented method of claim 1, wherein the determining a cost function includes:

for each potential depth value of a first location in the depth image, determining a cost for each potential depth value of each neighbor location, the cost is based on a depth difference between the potential depth value of the first location and the potential depth value of the neighbor location; and

repeating the determining a cost for each potential depth value of each neighbor location for each location in the depth image.

3. The machine-implemented method of claim 2, wherein the determining a cost function further includes:

determining an individual cost for each potential depth value for each of the locations in the depth image.

4. The machine-implemented method of claim 3, wherein the determining an individual cost for each potential depth value for each of the locations in the depth image includes assigning an individual cost to each potential depth value for a first location in the depth image based on a brightness value at the first location.

5. The machine-implemented method of claim 1, wherein the determining a cost function includes:

determining whether a pair of neighbor locations are likely to have about the same depth in a scene that corresponds to the depth image; and

penalizing large differences in potential depth values between pairs of neighbor locations that are determined to likely have about the same depth.

6. The machine-implemented method of claim 5, wherein the determining whether a pair of neighbor locations are likely to have about the same depth in a scene that corresponds to the depth image includes:

accessing a brightness image of the scene that corresponds to the depth image;

detecting edges in the brightness image; and

determining whether the pair of neighbor locations are likely to be at about the same depth based on the detected edges, the penalizing large differences in potential depth values between the pair of neighbor locations that are determined to likely have about the same depth includes increasing the cost of the pair of neighbor locations that are not near a strong edge, further comprising ignoring large differences in depth for pixels that are near a strong edge.

7. The machine-implemented method of claim 1, wherein accessing a depth image that includes a depth value for a plurality of locations includes:

accessing a depth value for a plurality of pixels in the depth image; and

dividing the depth image into a plurality of segments, each of the segments include a plurality of pixels having a neighbor with approximately the same depth value, each segment is a location of the plurality of locations.

8. The machine-implemented method of claim 1, wherein each of the plurality of locations corresponds to a pixel in the depth image.

9. An apparatus, comprising:

a processor; and

a computer readable storage medium coupled to the processor, the computer readable storage medium having instructions stored thereon which, when executed on the processor cause the processor to access a depth image that includes a depth value for a plurality of locations, each of the locations has one or more neighbor locations, the instructions further cause the processor to determine potential depth values for each of the plurality of locations based on the depth value in the depth image and potential aliasing in the depth image, the instructions further cause the processor to determine a cost function based on differences between the potential depth values of each location and its one or more neighbor locations, to determine the cost function the processor assigns a higher cost for greater differences in potential depth values between a pair of neighbor locations, the instructions further cause the processor to substantially minimize the cost function to select one of the potential depth values for each of the locations.

10. The apparatus of claim 9, wherein the instructions which cause the processor to determine a cost function cause the processor to determine a cost for each potential depth value of each neighbor location for each potential depth value of a first location in the depth image, the cost is based on a depth difference between the potential depth value of the first location of the potential depth value of the neighbor location, the instructions cause the processor determine a cost with each potential depth value of each neighbor location for each location in the depth image.

11. The apparatus of claim 10, wherein the instructions which cause the processor to determine a cost function further cause the processor to determine an individual cost for each potential depth value for each of the locations in the depth image.

12. The apparatus of claim 11, wherein the instructions which cause the processor to determine an individual cost for each potential depth value for each of the locations in the depth image cause the processor to assign an individual cost to each potential depth value for a first location in the depth image based on a brightness value at the first location.

13. The apparatus of claim 10, wherein the instructions which cause the processor to determine a cost function further cause the processor to determine whether a pair of neighbor locations are likely to have about the same depth in a scene that corresponds the depth image and to penalize large differences in potential depth values between the pair of neighbor locations that are determined to likely to have about the same depth.

14. The apparatus of claim 13, wherein the instructions further cause the processor to access a brightness image of the scene that corresponds to the depth image, detect edges in the brightness image, and determine whether the pair of neighbor locations are likely to be at about the same depth based on the detected edges, the instructions further cause the processor to increase the cost of the pair of neighbor locations that are not near a strong edge and to ignore large differences in depth values for pixels that are near a strong edge.

15. The apparatus of claim 10, wherein the instructions which cause the processor to access a depth image cause the processor to access a depth value for a plurality of pixels in the depth image, and divide the depth image into a plurality of segments, each of the segments include a plurality of pixels with a neighbor pixel having approximately the same depth value, each segment is a location of the plurality of locations.

16. The apparatus of claim 10, wherein each of the plurality of locations corresponds to a pixel in the depth image.

17. A machine-implemented method, comprising:

generating a depth image that includes a plurality of pixels, each pixel has one or more neighbor pixels, each pixel having a depth value, the depth image having a depth window;

determining potential depth values for each of the plurality of pixels based on the depth window and the depth value in the depth image for the pixel;

generating a graph that includes a plurality of layers of nodes, each pixel is represented by one node in each layer, the potential depth values for a given pixel are represented by nodes in different layers, pairs of nodes that correspond to neighbor pixels are connected by edges;

assigning a cost to each of the edges, the cost of a given edge is based on a depth difference between the potential depth values of the pair of nodes that are connected by the edge; and

reducing the graph to a single layer based on minimizing the costs of remaining edges, each of the pixels has one node in the reduced graph.

18. The machine-implemented method of claim 17, wherein generating the depth image includes:

modulating light at a first frequency;

transmitting the light onto an object in a field of view of a camera;

receiving light that reflects from the object;

comparing a phase of the modulated transmitted light with a phase of the received light; and

determining the depth image based on phase difference between the transmitted and received light and the first frequency.

19. The machine-implemented method of claim 17, further comprising:

determining a per-node cost for each of the nodes based on a brightness of the pixel that corresponds to the node, wherein the reducing the graph to a single layer substantially minimizes the sum of the costs of remaining edges and the per-node costs.

20. The machine-implemented method of claim 17, further comprising:

accessing a brightness image of a scene that corresponds to the depth image;

detecting edges in the visible image;

determining pairs of neighbor pixels that are likely to be at the same depth in the scene based on the detected edges;

for pairs of neighbor pixels that are likely to be at the same depth in the scene, determining which edges connect nodes that have a difference in potential depth values that is greater than a pre-determined threshold;

the assigning a cost to each of the edges includes increasing the cost of the edges that connect nodes that have a difference in potential depth values that is greater than a pre-determined threshold; and

for pairs of neighbor pixels that are not likely to be at the same depth in the scene, the assigning a cost to each of the edges includes decreasing the cost of the edges that connect nodes that have a difference in potential depth values that is greater than a pre-determined threshold.