HIGH-PERFORMANCE PLANE DETECTION WITH DEPTH CAMERA DATA

• US 20140363073A1
• Filed: 06/11/2013
• Published: 12/11/2014
• Est. Priority Date: 06/11/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shirakyan, Grigor, Jalobeanu, Mihai R.
• Application Number:
  US13/915,618
• Publication Number:
  US 20140363073A1
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 15/405 using Z-buffer
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The subject disclosure is directed towards detecting planes in a scene using depth data of a scene image, based upon a relationship between pixel depths, row height and two constants. Samples of a depth image are processed to fit values for the constants to a plane formulation to determine which samples indicate a plane. A reference plane may be determined from those samples that indicate a plane, with pixels in the depth image processed to determine each pixel'"'"'s relationship to the plane based on the pixel'"'"'s depth, location and associated fitted values, e.g., below the plane, on the plane or above the plane.

Claims

1. A method, comprising, processing depth data of an image to determine a plane, in which the depth data includes indexed rows and columns of pixels and a depth value for each pixel, including using a plurality of strips containing pixels, finding values for each strip that represent how well that strip'"'"'s pixels fit a plane formulation based upon depth values and pixel locations in the depth data corresponding to the strip, maintaining the values for at least some strips that indicate a plane based on whether the values meet an error threshold indicative of a plane, and associating sets of the maintained values with sets of pixels in the depth data.

2. The method of claim 1 wherein the plane is a reference plane, and wherein maintaining the values for at least some strips that indicate a plane comprises keeping the values for strips that correspond to the reference plane and not any other plane.

3. The method of claim 1 wherein the sets of pixels correspond to columns of pixels, and wherein associating the sets of the maintained values with the sets of pixels comprises associating a per-column set of the values with column of pixels.

4. The method of claim 3 further comprising, for a given pixel having a depth value, a column identifier and a row identifier in the depth data, using the depth value, the values associated with the pixel'"'"'s column, and the row identifier to estimate whether that pixel lies a) below the plane or above the plane, or b) on the plane, below the plane or above the plane.

5. The method of claim 3 further comprising using a change in one of the values across the columns to determine an amount of camera roll.

6. The method of claim 3 further comprising, interpolating a plurality of values corresponding to a column to find the values for associating with a selected column.

7. The method of claim 1 wherein the sets of values are determined for a frame, and further comprising reusing the constant values for a subsequent frame.

8. The method of claim 1 wherein finding the values for each strip comprises determining at least one of the values by iterative approximation.

9. The method of claim 1 wherein finding the values for each strip comprises determining at least one of the values by determining one of the constants by a binary search.

10. The method of claim 1 wherein the error threshold comprises a variable parameter, and further comprising, receiving the error threshold from an external source.

11. The method of claim 1 wherein processing the depth data of an image to determine a plane comprises determining a floor.

12. The method of claim 11 further comprising determining a substantially horizontal surface other than the floor based upon using the floor as a reference plane.

13. The method of claim 1 wherein processing the depth data of an image to determine a plane comprises determining a substantially vertical plane.

14. The method of claim 1 wherein using the strips comprises sampling a region with the plurality of strips.

15. The method of claim 1 wherein one of the values corresponds to a camera height relative to the plane, and wherein finding the values for each strip comprises constraining one of the values to be within a range of possible camera heights.

16. A system comprising, plane extraction logic configured to produce plane data for a scene, the plane extraction logic configured to input frames of depth data comprising pixels in which each pixel has a depth value, column index and row index, process the frame data to compute pairs of values for association with the pixels, in which for each pixel, a pair of values for the pixel, the depth value of the pixel, and the row or column index of the pixel indicate a relationship of that pixel to a reference plane.

17. The system of claim 16 wherein the reference plane is substantially horizontal, and wherein the pair of values for the pixel is associated with the pixel'"'"'s column index, and the pair of values for the pixel, the depth value of the pixel, and the row index of the pixel indicate a relationship of that pixel to the reference plane.

18. The system of claim 16 plane extraction logic processes the depth data by sampling strips of pixels to fit a pair of values to each strip.

19. One or more machine-readable storage media or logic having executable instructions, which when executed perform steps, comprising:

processing strips of pixel depth values, including for each strip, finding fitted values that fit a plane formula based upon row height and depth data for pixels of the strip;

eliminating the fitted values for any strip having pixels that do not correspond to a plane based upon a threshold evaluation that distinguishes planar strips from non-planar strips;

determining from non-eliminated strips which of the non-eliminated strips are likely on a reference plane; and

using the fitted values of the strips that are likely on the reference plane to associate a set of fitted values with each column of pixels.

20. The one or more machine-readable storage media or logic of claim 19 having further executable instructions comprising determining, for at least one pixel, a relationship between the pixel and the reference plane based upon the depth value of the pixel, a row height of the pixel and the set of fitted values associated with a column of the pixel.