Depth map movement tracking via optical flow and velocity prediction

• US 9,153,035 B2
• Filed: 10/02/2014
• Issued: 10/06/2015
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Finocchio, Mark J.
• Primary Examiner(s): Chauhan, Ulka
• Assistant Examiner(s): ZALALEE, SULTANA MARCIA
• Application Number:
  US14/504,949
• Publication Number:
  US 20150084967A1
• Time in Patent Office: 369 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Techniques for efficiently tracking points on a depth map using an optical flow are disclosed. In order to optimize the use of optical flow, isolated regions of the depth map may be tracked. The sampling regions may comprise a 3-dimensional box (width, height and depth). Each region may be “colored” as a function of depth information to generate a “zebra” pattern as a function of depth data for each sample. The disclosed techniques may provide for handling optical flow tracking when occlusion occurs by utilizing a weighting process for application of optical flow vs. velocity prediction to stabilize tracking.

Claims

1. A computer implemented method for movement tracking, comprising:

a computing system processing a depth map using an optical flow algorithm to determine an optical flow vector for the depth map, the depth map comprising a point that indicates a depth value;

the computing system generating velocity prediction data for the point;

the computing system analyzing the depth map to determine a value representative of obfuscation of the depth map;

the computing system determining a new location for the point based on a weighting of the optical flow vector and the velocity prediction data, a weight given to the velocity prediction data being increased relative to a weight given to the optical flow vector as the determined value representative of obfuscation increases; and

the computing system setting a location of the point as the new location.

2. The computer implemented method of claim 1, wherein processing a depth map using an optical flow algorithm to determine an optical flow vector for the depth map comprises processing a depth image using a Lucas-Kanade algorithm.

3. The computer implemented method of claim 1, wherein analyzing the depth map to determine a value representative of obfuscation of the depth map comprises analyzing the depth map to determine a confidence value.

4. The computer implemented method of claim 3, wherein analyzing the depth map to determine a confidence value comprises analyzing the depth map to determine a confidence value indicating the degree to which optical flow processing may be applied to the depth map.

5. The computer implemented method of claim 1, wherein analyzing the depth map to determine a value representative of obfuscation of the depth map comprises analyzing the depth map to determine a value indicating the depth map is classified good, obfuscated, or unknown.

6. The computer implemented method of claim 1, wherein the computing system generating velocity prediction data for the point comprises generating velocity prediction data from recently known velocities.

7. The computer implemented method of claim 4, wherein the computing system determining a new location for the point based on a weighting of the optical flow vector and the velocity prediction data comprises determining a new location for the point by performing a weighting of the optical flow tracking vector and the velocity prediction based on the confidence value.

8. The computer implemented method of claim 1, further comprising the computing system assigning a gray scale value to the point as a function of the depth value.

9. The computer implemented method of claim 8, wherein processing a depth map using an optical flow algorithm to determine an optical flow vector for the depth map comprises processing the gray scale value.

10. A system for movement tracking, comprising:

a computing processor; and

computing memory communicatively coupled to the computing processor, the computing memory having stored therein computer executable instructions that when executed cause the system to perform operations comprising;

processing a depth map using an optical flow algorithm to determine an optical flow vector for the depth map, the depth map comprising a point that indicates a depth value, the depth map representing a position of an object with respect to a depth camera;

generating velocity prediction data for the point;

analyzing the depth map to determine a value representative of obfuscation of the depth map; and

determining a new location for the point based on a weighting of the optical flow vector and the velocity prediction data, a weight given to the velocity prediction data being increased relative to a weight given to the optical flow vector as the determined value representative of obfuscation increases.

11. The system of claim 10, the computing memory having stored therein computer executable instructions that when executed cause the system to perform operations further comprising:

receiving depth map data;

generating a region of depth map data comprising the point that indicates a depth value.

12. The system of claim 11, wherein generating a region of depth map data comprises generating a three-dimensional volume oriented around the point.

13. The system of claim 11, wherein generating a region of depth map data comprising the point that indicates a depth value comprises filtering data outside of a predefined region.

14. The system of claim 11, wherein processing a depth map using an optical flow algorithm to determine an optical flow vector for the depth map comprises processing the generated region of depth map data comprising the point that indicates a depth value.

15. The system of claim 14, the computing memory having stored therein computer executable instructions that when executed cause the system to perform operations further comprising:

assigning a gray scale value to the point as a function of the depth value,

wherein processing the generated region comprising the point that indicates a depth value comprises processing the assigned gray scale value.

16. The system of claim 10, the computing memory having stored therein computer executable instructions that when executed cause the system to perform operations further comprising:

determining a user input to the system based on the new location.

17. The system of claim 10, further comprising a depth camera, the depth camera generating depth map data representing positions of a system user with respect to the depth camera.

18. The system of claim 10, wherein the processor and computing memory are comprised in a game console.

19. The system of claim 18, wherein the game console is a video game console.

20. A computer readable storage device having stored therein computer executable instructions that when executed cause a computing system to perform operations comprising:

receiving depth map data;

identifying regions from the depth map data, each identified region comprising a selected point indicating a depth value;

generating for each of the identified regions gray scale values for a selected point;

processing generated gray scale values for each of the identified regions using an optical flow algorithm to generate optical flow vectors;

generating velocity prediction data for a selected point in each identified region;

analyzing each identified region to determine values representative of obfuscation;

determining new locations for points based on a weighting of optical flow vectors and velocity prediction data, weights given to velocity prediction data increasing relative to weights given to optical flow vectors as determined values representative of obfuscation increase.