Systems and methods of tracking object movements in three-dimensional space

• US 9,105,103 B2
• Filed: 01/23/2014
• Issued: 08/11/2015
• Est. Priority Date: 01/23/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David, Hall, W. Dale
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): DELANEY, MOLLY K
• Application Number:
  US14/162,720
• Publication Number:
  US 20140205146A1
• Time in Patent Office: 565 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

The technology disclosed relates to tracking movement of a real world object in three-dimensional (3D) space. In particular, it relates to mapping, to image planes of a camera, projections of observation points on a curved volumetric model of the real world object. The projections are used to calculate a retraction of the observation points at different times during which the real world object has moved. The retraction is then used to determine translational and rotational movement of the real world object between the different times.

Claims

1. A method of tracking movement of an object portion in three-dimensional (3D) space, the method including:

selecting five observation points on a curved volumetric model fitted to a portion of a real world object, wherein one of the observation points is at a center of the curved volumetric model and the other four observation points are along a perimeter of the curved volumetric model;

capturing at times t0 and t1 projections of at least the four perimeter observation points in at least a first image plane of at least one camera, wherein the object moved between t0 and t1;

calculating a retraction of the four perimeter observation points at time t1 to their positions at t0, including using the four perimeter observation points to;

map the captured projections of the perimeter observation points at t0 to a first image plane; and

calculate orientation of a second image plane such that the captured projections of the perimeter observation points at t1 to the second image plane are collinear with respective projections in the first image plane with respect to the retracted center observation point; and

determining at least translation of the curved volumetric model between the times t0 and t1 using the calculated orientation of the second image plane.

2. The method of claim 1, further including:

selecting a sixth observation point along a perimeter of the curved volumetric model;

capturing at the times t0 and t1 projections of the sixth observation point in a third image plane of the camera; and

using a difference in position of the sixth observation point in the third image plane combined with the calculated orientation of the second image plane to calculate rotation of the curved volumetric model.

3. The method of claim 1, further including calculating the retraction by rescaling positions of the perimeter observation points based on distances between the captured projections of the perimeter observation points at the times t0 and t1 and corresponding centers of the curved volumetric model at the times t0 and t1.

4. The method of claim 1, further including calculating the refraction using projective transformations of a mapping matrix and an inverse mapping matrix, including:

applying a canonical frame to projections of the perimeter observation points at t0 mapped to a first image plane and projections of the perimeter observation points at t1 mapped to a second image plane; and

applying a forward initial transformation of the projections of the perimeter observation points at t0 to inverse initial transformation projections of the perimeter observation points at t1.

5. The method of claim 4, further including constructing the mapping matrix by:

mapping captured projections of the perimeter observation points at t0 to captured projections of the perimeter observation points at t1, including;

extending a line connecting a point on the first image plane with a center of the curved volumetric model at time t0; and

intersecting the extended line with the second image plane.

6. The method of claim 4, further including constructing the inverse mapping matrix by:

mapping captured projections of the perimeter observation points at t1 to captured projections of the perimeter observation points at t0, including;

extending a line connecting a point on the second image plane; and

intersecting the extended line with the second image plane.

7. The method of claim 4, further including determining horizon lines on the first and second image planes that are mapped to infinity based on the projective transformations of the mapping matrix and inverse mapping matrix.

8. The method of claim 4, further including determining positions of the centers of the curved volumetric model at the times t0 and t1 by identifying an intersection line between the first and second image planes and corresponding vanishing planes not mapped to any image plane.

9. The method of claim 8, further including identifying the intersection line based on geometry of the first and second image planes and corresponding vanishing planes by:

determining planes lines that include orthogonal projections of a center of the curved volumetric model in the first and second image planes and are parallel to horizon lines in the first and second image planes;

extending projection lines that pass through the center of the curved volumetric model and are orthogonal to the plane lines;

responsive to extension of an auxiliary line in a third image plane that passes through the center of the curved volumetric model and is parallel to the horizon lines in the first and second image planes, creating a pair of intersection lines where the third image plane intersects with the first and second image planes; and

responsive to construction of congruent triangles by the pair of intersection lines and the horizon lines in the first and second image planes, calculating orthogonal distance between the horizon lines in the first and second image planes and the intersection line between the first and second image planes.

10. The method of claim 2, further including determining a range of values of an angle between the first and second image planes based on the sixth observation point along the perimeter of the curved volumetric model.

11. The method of claim 8, wherein determining positions of the centers of the curved volumetric model at the times t0 and t1 further includes:

constructing two axes that intersect at two points and also intersect the intersection line based on the mapping matrix and inverse mapping matrix, geometry of the first and second image planes and corresponding vanishing planes, and a range of values of the angle between the first and second image planes.

12. The method of claim 10, further including determining a relative rotational angle of the curved volumetric from t0 to t1 based on the captured projections of the perimeter observation points at the times t0 and t1 and positions of the centers of the curved volumetric model for the range of values of the angle between the first and second image planes.

13. The method of claim 10, further including:

calculating a mapping error for the range of values of the angle by comparing new curved volumetric models constructed based on the determined positions of the centers of the curved volumetric model with the observation points along the perimeter of the curved volumetric model; and

selecting a particular new curved volumetric model with minimum mapping error.

14. The method of claim 13, further including calculating positions of the observation points by normalizing, to a unit length, radii directed from corresponding centers of the new curved volumetric models to projections of the observations points.

15. A method of tracking movement of an object portion in three-dimensional (3D) space, the method including:

capturing at times t0 and t1 projections of at least four perimeter observation points on a curved volumetric model fitted to a portion of a real world object;

calculating a retraction of the four perimeter observation points at time t1 to their positions at t0 by;

using projective transformations of a mapping matrix and an inverse mapping matrix based on a first image plane to which projections of the perimeter observation points at t0 are mapped and a second image plane to which projections of the perimeter observation points at t1 are mapped;

determining horizon lines on the first and second image planes that are mapped to infinity based on the projective transformations of the mapping matrix and inverse mapping matrix;

determining positions of centers of the curved volumetric model at the times t0 and t1 by identifying an intersection line between the first and second image planes and corresponding vanishing planes not mapped to any image plane;

determining a range of values of an angle between the first and second image planes based on a sixth observation point along the perimeter of the curved volumetric model;

determining positions of the centers of the curved volumetric model at the times t0 and t1 by constructing two axes that intersect at two points and also intersect the intersection line based on the mapping matrix and inverse mapping matrix, geometry of the first and second image planes and corresponding vanishing planes, and the range of values of the angle between the first and second image planes; and

determining a relative rotational angle of the curved volumetric from t0 to t1 based on the captured projections of the perimeter observation points at the times t0 and t1 and positions of the centers of the curved volumetric model for the range of values of the angle between the first and second image planes.

16. The method of claim 15, further including:

calculating a mapping error for the range of values of the angle by comparing new curved volumetric models constructed based on the determined positions of the centers of the curved volumetric model with the observation points along the perimeter of the curved volumetric model; and

selecting a particular new curved volumetric model with minimum mapping error.

17. The method of claim 15, further including calculating positions of the observation points by normalizing, to a unit length, radii directed from the corresponding centers of the new curved volumetric models to projections of the observations points.