Three dimensional (3D) modeling of a complex control object

• US 9,646,201 B1
• Filed: 06/05/2015
• Issued: 05/09/2017
• Est. Priority Date: 06/05/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Horowitz, Kevin A.
• Primary Examiner(s): Heidemann, Jason
• Assistant Examiner(s): SHIN, BRIAN D
• Application Number:
  US14/732,616
• Time in Patent Office: 704 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

The disclosed technology automatically (e.g., programmatically) initializes predictive information for tracking a complex control object (e.g., hand, hand and tool combination, robot end effector) based upon information about characteristics of the object determined from sets of collected observed information. Automated initialization techniques obviate the need for special and often bizarre start-up rituals (place your hands on the screen at the places indicated during a full moon, and so forth) required by conventional techniques. In implementations, systems can refine initial predictive information to reflect an observed condition based on comparison of the observed with an analysis of sets of collected observed information.

Claims

1. A method of initializing predictive information that models a complex control object in a three dimensional (3D) sensory space, the method including:

accessing observed information including a set of contour points corresponding to surface points at along an outline of a complex control object in a three dimensional (3D) sensory space;

transforming the set of contour points to a normalized orientation of the control object, including;

at training time t0, sensing an actual position of at least one complex control object in a first reference frame of the 3D sensory space;

at initialization time t1, sensing, in the 3D sensory space, an apparent position of the complex control object different from the actual position, wherein the complex control object has not moved in the 3D sensory space between t0 and t1;

calculating a second reference frame that accounts for apparent position of the complex control object;

calculating a transformation that renders the actual position in the first reference frame and the apparent position in the second reference frame into a common reference frame; and

transforming the actual and apparent positions of the complex control object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed apparent position is transformed to an actual position;

searching a plurality of observed information archetypes that represent poses of the control object in the normalized orientation, the poses including arrangement of features of the complex control object and a perspective of observing the complex control object, and selecting an archetype; and

initializing predictive information that models the complex control object from initialization parameters associated with the selected archetype.

2. The method of claim 1, wherein the common reference frame is a world reference frame that does not change.

3. The method of claim 1, wherein the transforming the actual and apparent positions of the complex control object into the common reference frame further includes applying an affine transformation.

4. The method of claim 1, further including determining the orientation of the complex control object at the actual position with respect to the first reference frame.

5. The method of claim 1, further including determining the orientation of the complex control object at the apparent position with respect to the second reference frame.

6. The method of claim 1, further including determining a position of the complex control object at the actual position by calculating a translation of the complex control object with respect to the common reference frame.

7. The method of claim 1, further including determining a position of the complex control object at the apparent position by calculating a translation of the complex control object with respect to the common reference frame.

8. The method of claim 1, wherein the transforming further includes at least one of:

applying a vector to the set of contour points; and

applying a rotation matrix to the set of contour points.

9. The method of claim 1, wherein the searching further includes traversing a linked data structure including the plurality of observed information archetypes.

10. The method of claim 9, wherein the traversing further includes:

visiting a node in the data structure;

comparing the transformed contour points sets to one or more pluralities of observed information archetypes associated with the node; and

selecting, from the pluralities, at least one archetype having highest conformance with the transformed contour points sets of the control object.

11. The method of claim 9, wherein the linked data structure includes a plurality of nodes representing observed information archetypes in parent-child relationship and the traversing further includes:

visiting a plurality of parent nodes, each parent node in the plurality identifying one or more variants of one or more poses, and calculating a ranked list of parent nodes having highest conformance with the transformed contour points sets of the control object; and

visiting a plurality of child nodes related to the parent nodes in the ranked list, each child node identifying one or more variants of one or more poses different from the one or more poses of the parent nodes, and calculating a ranked list of child nodes having highest conformance with the transformed contour points sets of the control object.

12. The method of claim 1, wherein the initializing predictive information further includes:

aligning one or more model portions based at least in part upon one or more initialization parameters associated with the selected archetype.

13. The method of claim 1, wherein the complex control object is a hand and the initialization parameters include:

edge information for at least fingers of the hand.

14. The method of claim 1, wherein the complex control object is a hand and the initialization parameters include:

edge information for a palm of the hand.

15. The method of claim 1, wherein the complex control object is a hand and the initialization parameters include:

finger segment length information for fingers of the hand.

16. The method of claim 1, wherein the complex control object is a hand and the initialization parameters include:

at least one of;

one or more joint angles between finger segments of fingers of the hand;

a pitch angle between finger segments of fingers of the hand; and

a yaw angle between finger segments of fingers of the hand.

17. The method of claim 1, wherein the complex control object is a hand and the initialization parameters include:

joint angle and segment orientation information of the hand.

18. The method of claim 1, wherein the complex control object is a hand and the initialization parameters include:

a distance between adjoining base points of fingers of the hand.

19. The method of claim 1, wherein the complex control object is a hand and the initialization parameters include:

a ratio of distance between adjoining base points of fingers of the hand to minimal distance between adjoining base points of the fingers.

20. The method of claim 1, wherein the complex control object is a hand and the poses identify:

an angle between adjacent fingers of the hand.

21. The method of claim 1, wherein the complex control object is a hand and the poses identify:

a joint angle between adjacent finger segments of the hand.

22. The method of claim 1, wherein the complex control object is a hand and the poses identify:

a ratio of hand'"'"'s fingers'"'"' thickness to a maximal finger'"'"'s thickness.

23. The method of claim 1, wherein the complex control object is a hand and the poses identify:

span lengths between opposing sides of the hand.

24. The method of claim 1, wherein the complex control object is a hand and the poses identify:

at least one of;

finger diameter length fingers of the hand;

palm length of palm of the hand;

palm to thumb distance of the hand;

wrist length of wrist of the hand; and

wrist width of wrist of the hand.

25. The method of claim 1, wherein the complex control object is a hand and further including:

using the selected archetype to determine at least one of;

whether one or more fingers of the hand are extended or non-extended;

one or more angles of bend for one or more fingers;

a direction to which one or more fingers point; and

a configuration indicating a pinch, a grab, an outside pinch, or a pointing finger.

26. The method of claim 1, wherein the complex control object is an automobile and the initialization parameters include:

at least one of;

cabin of the automobile;

windshield to rear distance of the automobile;

front bumper to rear bumper distance of the automobile; and

distance between front of a tire and rear of the tire of the automobile.

27. A system of initializing predictive information that models a complex control object in a three dimensional (3D) sensory space, the system including:

a camera; and

one or more processors coupled to the camera and to a non-transitory computer readable medium storing instructions thereon, which instructions when executed by the processors perform;

accessing observed information including a set of contour points corresponding to surface points at along an outline of a complex control object in a three dimensional (3D) sensory space;

transforming the set of contour points to a normalized orientation of the control object, including;

at training time t0, sensing an actual position of at least one complex control object in a first reference frame of the 3D sensory space;

at initialization time t1, sensing, in the 3D sensory space, an apparent position of the complex control object different from the actual position, wherein the complex control object has not moved in the 3D sensory space between t0 and t1;

calculating a second reference frame that accounts for apparent position of the complex control object;

calculating a transformation that renders the actual position in the first reference frame and the apparent position in the second reference frame into a common reference frame; and

transforming the actual and apparent positions of the complex control object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed apparent position is transformed to an actual position;

searching a plurality of observed information archetypes that represent poses of the control object in the normalized orientation, the poses including arrangement of features of the complex control object and a perspective of observing the complex control object, and selecting an archetype; and

initializing predictive information that models the complex control object from initialization parameters associated with the selected archetype.

28. A non-transitory computer readable memory containing computer program instructions for initializing predictive information that models a complex control object in a three dimensional (3D) sensory space, which instructions when executed by one or more processors perform:

accessing observed information including a set of contour points corresponding to surface points at along an outline of a complex control object in a three dimensional (3D) sensory space;

transforming the set of contour points to a normalized orientation of the control object, including;

at training time t0, sensing an actual position of at least one complex control object in a first reference frame of the 3D sensory space;

at initialization time t1, sensing, in the 3D sensory space, an apparent position of the complex control object different from the actual position, wherein the complex control object has not moved in the 3D sensory space between t0 and t1;

calculating a second reference frame that accounts for apparent position of the complex control object;

calculating a transformation that renders the actual position in the first reference frame and the apparent position in the second reference frame into a common reference frame; and

transforming the actual and apparent positions of the complex control object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed apparent position is transformed to an actual position;

searching a plurality of observed information archetypes that represent poses of the control object in the normalized orientation, the poses including arrangement of features of the complex control object and a perspective of observing the complex control object, and selecting an archetype; and

initializing predictive information that models the complex control object from initialization parameters associated with the selected archetype.