DEPTH MAP MOVEMENT TRACKING VIA OPTICAL FLOW AND VELOCITY PREDICTION

• US 20100194741A1
• Filed: 01/30/2009
• Published: 08/05/2010
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Finocchio, Mark J.
• Granted Patent: US 8,866,821 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Techniques for efficiently tracking points on a depth map using an optical flow are disclosed. In order to optimize the use of optical flow, isolated regions of the depth map may be tracked. The sampling regions may comprise a 3-dimensional box (width, height and depth). Each region may be “colored” as a function of depth information to generate a “zebra” pattern as a function of depth data for each sample. The disclosed techniques may provide for handling optical flow tracking when occlusion occurs by utilizing a weighting process for application of optical flow vs. velocity prediction to stabilize tracking.

Claims

1. A method for performing optical flow on a depth map comprising a plurality of points, each point indicating a depth value, the method comprising:

for each of at least one selected point in the depth map, determining an associated region comprising at least two of the points in the depth map including the respective selected point;

processing each region by assigning a grayscale value to each point in the region as a function of each point'"'"'s associated depth value;

processing each region using an optical flow algorithm to determine optical flow vectors for the region;

for each selected point, using the optical flow vectors for the associated region that includes the selected point to determine a new point location; and

setting each selected point to a respective new point.

2. The method of claim 1, wherein each region comprises a three-dimensional volume oriented around the respective selected point

3. The method of claim 2, wherein the three-dimensional volume filters out points beyond a selected depth.

4. The method of claim 1, further comprising performing a velocity prediction for each point in a region.

5. The method of claim 4, further comprising determining a confidence value for each region.

6. The method of claim 5, wherein the optical flow vectors are computed based upon a weighting of the optical flow algorithm and the velocity prediction, the weighting controlled by the confidence value.

7. The method of claim 1, wherein the optical flow algorithm is the Lucas-Kanade algorithm.

8. A computer readable medium having stored thereon computer executable instructions for performing a process for determining user input to a computer based on optical flow, the process comprising:

receiving depth map data representing positions of a computer user with respect to a depth camera;

performing optical flow on the depth map data to track movement of the user; and

determining a user input to the computer based on the calculated movement.

9. The computer readable medium of claim 8, wherein the computer is a game console.

10. The computer readable medium of claim 8, wherein the user is a video game player.

11. The computer readable medium of claim 8, wherein performing optical flow comprises:

for each of at least one selected point in a depth map, determining an associated region comprising at least two of the points in the depth map including the respective selected point;

processing each region by assigning a grayscale value to each point in the region as a function of each point'"'"'s associated depth value;

processing each region using an optical flow algorithm to determine optical flow vectors for the region;

for each selected point, using the optical flow vectors for the associated region that includes the selected point to determine a new point location; and

setting each selected point to a respective new point.

each region comprises a three-dimensional volume oriented around the respective selected point

12. The computer readable medium of claim 9, wherein the process further comprises performing a velocity prediction for each point in a region.

13. The computer readable medium of claim 12, wherein the instructions further comprise determining a confidence value for each region.

14. The computer readable medium of claim 13, wherein the optical flow vectors are computed based upon a weighting of the optical flow algorithm and the velocity prediction, the weighting controlled by the confidence value.

15. A system for performing optical flow on depth map data comprising:

a point selection module, the point selection module receiving a point selection input comprising at least one point to be tracked using optical flow and a depth map;

a sampling module that generates at least one region data element, each region data element corresponding to a respective point and comprising filtered depth map data;

a depth map processing module that processes each region data element to generate a respective colored region data element comprising a plurality of values as a function of the depth map data for the region data element;

a velocity prediction component that predicts a velocity of each point;

a confidence module that generates a confidence value of each region data element; and

an optical flow tracking module that, for each region data element, is adapted to;

process the region data element using an optical flow algorithm to determine optical flow vectors for the corresponding point; and

determine an updated position of the corresponding point by weighting the optical flow vectors as a function of the velocity for the point and the confidence value for the region.

16. The system of claim 15, wherein each region data element comprises a three-dimensional volume oriented around the corresponding point

17. The method of claim 16, wherein the three-dimensional volume filters out points beyond a selected depth.

18. The system of claim 15, wherein the optical flow vectors are computed based upon a weighting of the optical flow algorithm and the velocity prediction, the weighting controlled by the confidence value.

19. The system of claim 15, wherein the optical flow algorithm is the Lucas-Kanade algorithm.

20. The system of claim 15, wherein the colored region data values are grayscale values.