Invariant features for computer vision

• US 8,878,906 B2
• Filed: 11/28/2012
• Issued: 11/04/2014
• Est. Priority Date: 06/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shotton, Jamie D. J., Finocchio, Mark J., Moore, Richard E., Balan, Alexandru O., Lee, Kyungsuk David
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Zhou, Zhihan
• Application Number:
  US13/688,120
• Publication Number:
  US 20140002607A1
• Time in Patent Office: 706 Days
• Field of Search: 348/46, 382/154, 382/103, 382/181, 382/190, 382/195, 382/209, 382/224
• US Class Current: 348/46
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

Technology is described for determining and using invariant features for computer vision. A local orientation may be determined for each depth pixel in a subset of the depth pixels in a depth map. The local orientation may an in-plane orientation, an out-out-plane orientation or both. A local coordinate system is determined for each of the depth pixels in the subset based on the local orientation of the corresponding depth pixel. A feature region is defined relative to the local coordinate system for each of the depth pixels in the subset. The feature region for each of the depth pixels in the subset is transformed from the local coordinate system to an image coordinate system of the depth map. The transformed feature regions are used to process the depth map.

Claims

1. A method, comprising:

accessing a depth map that includes a plurality of depth pixels, the depth map is associated with an image coordinate system having a plane;

estimating a local orientation for each depth pixel in a subset of the depth pixels, the local orientation includes an in-plane orientation relative to the plane of the image coordinate system, determining the local orientation for the in-plane orientation comprises;

determining a closest point between a test depth pixel and a reference line of an object in the depth map, and determining a rotation invariant angle for the test depth pixel based on an angle between a reference axis and a line between the test depth pixel and the closest point;

defining a local coordinate system for each of the depth pixels in the subset, each local coordinate system is based on the local orientation of the corresponding depth pixel;

defining a feature region relative to the local coordinate system for each of the depth pixels in the subset;

transforming the feature region for each of the depth pixels in the subset from the local coordinate system to the image coordinate system; and

using the transformed feature regions to process the depth map.

2. The method of claim 1, wherein the using the transformed feature regions to process the depth map includes:

classifying features from the depth map based on the transformed feature regions.

3. The method of claim 1, wherein the using the transformed feature regions to process the depth map includes:

training a machine learning classifier based on the transformed feature regions.

4. The method of claim 1, wherein the local orientation includes an out-of-plane orientation relative to the plane of the image coordinate system, wherein the estimating a local orientation includes estimating an out-of-plane rotation, the out-of-plane rotation is defined by a surface normal to the object in the depth map at a test depth pixel.

5. The method of claim 4, wherein the defining a local coordinate system for each of the depth pixels in the subset includes defining a z-axis of the local coordinate system as being the surface normal.

6. The method of claim 1, wherein the local orientation includes an out-of-plane orientation relative to the plane of the image coordinate system, wherein the defining a local coordinate system for each of the depth pixels in the subset includes:

determining a vector in 3D space from a test depth pixel to a point of interest;

determining a surface normal to the object in the depth map; and

establishing an x-axis that is the component of the vector in 3D space that is orthogonal to the surface normal.

7. The method of claim 6, wherein the defining a local coordinate system for each of the depth pixels in the subset includes establishing a y-axis as the cross product of a z-axis and the x-axis.

18. The method of claim 1, wherein the reference line of the object comprises an edge or a medial axis.

8. A system comprising:

a depth camera for generating depth maps that includes a plurality of depth pixels, each pixel having a depth value, each depth map is associated with a 2D image coordinate system;

logic coupled to the depth camera, the logic is operable to;

access a depth map from the depth camera, the depth map is associated with an image coordinate system having a plane;

estimate a local orientation for each depth pixel in a subset of the depth pixels, the local orientation includes an in-plane orientation that is in the plane of the 2D image coordinate system, when the logic determines the local orientation based on the in-plane orientation the logic;

determines a closest point between a test depth pixel and a reference line of an object in the depth map; and

determines a rotation invariant angle for the test depth pixel based on an angle between a reference axis and a line between the test depth pixel and the closest point;

define a local 3D coordinate system for each of the depth pixels in the subset, each local 3D coordinate system is based on the local orientation of the corresponding depth pixel;

define a feature region relative to the local coordinate system for each of the depth pixels in the subset;

transform the feature region for each of the depth pixels in the subset from the local 3D coordinate system to the 2D image coordinate system; and

identify the object in the depth map based on the transformed feature regions.

9. The system of claim 8, wherein the local orientation includes an out-of-plane orientation that is out of the plane of the 2D coordinate system, wherein when the logic estimates a local orientation the logic estimates an out-of-plane rotation, the out-of-plane rotation is defined by a surface normal to an object in the depth map at a test depth pixel.

10. The system of claim 9, wherein when the logic defines a local coordinate system for each of the depth pixels in the subset the logic defines a z-axis of the local coordinate system as being the surface normal.

11. The system of claim 8, wherein the local orientation includes an out-of-plane orientation that is out of the plane of the 2D coordinate system, wherein when the logic defines a local 3D coordinate system for each of the depth pixels in the subset the logic:

determines a vector in 3D space from a test depth pixel to a point of interest;

determines a surface normal to the object in the depth map; and

establishes an x-axis that is the component of the vector in 3D space that is orthogonal to the surface normal.

12. The system of claim 11, wherein when the logic defines a local coordinate system for each of the depth pixels in the subset the logic establishes a y-axis as the cross product of a z-axis and the x-axis.

13. A computer readable storage device having instructions stored thereon which, when executed on a processor, cause the processor to:

access a depth map that includes an array of depth pixels, each depth pixel has a depth value, the depth map is associated with a 2D image coordinate system;

estimate a local orientation for each depth pixel in a subset of the depth pixels, the local orientation includes in-plane orientation that is in the plane of the 2D image coordinate system and an out-of-plane orientation that is out-of-the plane of the 2D image coordinate system, when the processor determines the local orientation based on the in-plane orientation the processor;

determines a closest point between a test depth pixel and a reference line of an object in the depth map, and the processor determines a rotation invariant angle for the test depth pixel based on an angle between a reference axis and a line between the test depth pixel and the closest point;

determine a 3D model for the depth map, the model includes a plurality of 3D points that are based on the depth pixels, each of the points has a corresponding depth pixel;

define a local 3D coordinate system for each of the plurality of points, each local 3D coordinate system is based on a position and the local orientation of the corresponding depth pixel;

define feature test points relative to the local coordinate system for each of the points;

transform the feature test points from the local 3D coordinate system to the 2D image coordinate system for each of the feature test points; and

identify the object in the depth map based on the transformed feature test points.

14. The computer readable storage device of claim 13, wherein the instructions that cause the processor to transform the feature test points cause the processor to rotate the feature test points using a rotation matrix R=[{right arrow over (r₁)}, {right arrow over (r₂)}, {right arrow over (r₃)}], where {right arrow over (r₃)} is a unitized surface normal to the object, {right arrow over (r₁)} is the component of an in-plane rotation-variant vector that is orthogonal to the surface normal, and {right arrow over (r₂)} is the cross product between {right arrow over (r₃)} and {right arrow over (r₁)}.

15. The computer readable storage device of claim 13, wherein the out-of-plane rotation is defined by a surface normal to the object at the test depth pixel, the instructions that cause the processor to define a local coordinate system for each of the plurality of points cause the processor to define a z-axis of the local coordinate system as being the surface normal.

16. The computer readable storage device claim 13, wherein the instructions that cause the processor to define a local coordinate system for each of the plurality of points cause the processor to:

determine a vector in 3D space from a first of the points to a point of interest;

determine a surface normal to the object; and

establish an x-axis that is the component of the vector in 3D space that is orthogonal to the surface normal.

17. The computer readable storage device of claim 16, wherein the instructions that cause the processor to define a local coordinate system for each of the plurality of points cause the processor to establish a y-axis as the cross product of a z-axis and the x-axis.