Depth map movement tracking via optical flow and velocity prediction

• US 8,866,821 B2
• Filed: 01/30/2009
• Issued: 10/21/2014
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Finocchio, Mark J.
• Primary Examiner(s): Chauhan, Ulka
• Assistant Examiner(s): ZALALEE, SULTANA MARCIA
• Application Number:
  US12/363,036
• Publication Number:
  US 20100194741A1
• Time in Patent Office: 2,090 Days
• Field of Search: 345/473
• US Class Current: 345/473
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Techniques for efficiently tracking points on a depth map using an optical flow are disclosed. In order to optimize the use of optical flow, isolated regions of the depth map may be tracked. The sampling regions may comprise a 3-dimensional box (width, height and depth). Each region may be “colored” as a function of depth information to generate a “zebra” pattern as a function of depth data for each sample. The disclosed techniques may provide for handling optical flow tracking when occlusion occurs by utilizing a weighting process for application of optical flow vs. velocity prediction to stabilize tracking.

Claims

1. A method for performing optical flow on a depth map comprising a selected point and a second point, each point indicating a depth value, the method comprising:

determining region comprising the selected and second points;

processing the region by assigning a grayscale value to each point as a function of each point'"'"'s associated depth value;

processing the region using an optical flow algorithm to determine optical flow vectors for the region;

performing a velocity prediction of the selected point;

analyzing the region to determine an amount of occlusion of the region;

determining a new location for the selected point based on a weighting of the optical flow vectors for the region and the velocity prediction, the weighting being based on the determined amount of occlusion in the region such that a relative weight given to the velocity prediction and a relative weight given to the optical flow vectors depends on the determined amount of occlusion, and the weight given to the velocity prediction is increased relative to the weight given to the optical flow vectors as the determined amount of occlusion increases; and

setting the location of the selected point to the new location.

2. The method of claim 1, wherein the region comprises a three-dimensional volume oriented around the selected point.

3. The method of claim 2, wherein the three-dimensional volume filters out points beyond a selected depth.

4. The method of claim 1, wherein the optical flow algorithm is the Lucas-Kanade algorithm.

5. A computer-readable storage device storing optical flow bearing computer-executable instructions used for performing a process for determining user input to a computer that, when executed on the computer, cause the computer to perform operations comprising:

receiving depth map data representing positions of a computer user with respect to a depth camera;

determining an associated region comprising a selected point and a second point in the depth map;

determining a grayscale value for the selected and second points based on a respective depth value of that point;

performing optical flow on the depth map data to determine optical flow vectors for the region;

performing a velocity prediction of the selected point;

analyzing the region to determine an amount of occlusion of the region;

determining a new location for the selected point based on a weighting of the optical flow vectors for the associated region and the velocity prediction, the weighting being based on the determined amount of occlusion in the associated region, and the weight given to the velocity prediction is increased relative to the weight given to the optical flow vectors as the determined amount of occlusion increases;

setting the location of the selected point to the new location; and

determining a user input to the computer based on the new location of the selected point.

6. The computer-readable storage device of claim 5, wherein the computer is a game console.

7. The computer-readable storage device of claim 5, wherein the user is a video game player.

8. The computer-readable storage device of claim 5, wherein the region comprises a three-dimensional volume oriented around the respective selected point.

9. A system for performing optical flow on depth map data comprising a selected point, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system at least to;

determine an associated region comprising the selected and second points;

process the region by assigning a grayscale value to each point as a function of each point'"'"'s associated depth value;

process the region using an optical flow algorithm to determine optical flow vectors for the region;

perform a velocity prediction of the selected point;

analyze the region to determine an amount of occlusion of the region;

determine a new location for the selected point based on a weighting of the optical flow vectors for the associated region and the velocity prediction, the weighting being based on the determined amount of occlusion in the associated region, and the weight given to the velocity prediction is increased relative to the weight given to the optical flow vectors as the determined amount of occlusion increases; and

set the location of the selected point to the new location.

10. The system of claim 9, wherein the optical flow algorithm is the Lucas-Kanade algorithm.