Predictive information for free space gesture control and communication

• US 9,996,638 B1
• Filed: 10/31/2014
• Issued: 06/12/2018
• Est. Priority Date: 10/31/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S, Bedikian, Raffi, Horowitz, Kevin, Yang, Hua
• Primary Examiner(s): Ho, Andy
• Application Number:
  US14/530,690
• Time in Patent Office: 1,320 Days
• Field of Search: 703 2
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed relates to simplifying updating of a predictive model using clustering observed points. In particular, it relates to observing a set of points in 3D sensory space, determining surface normal directions from the points, clustering the points by their surface normal directions and adjacency, accessing a predictive model of a hand, refining positions of segments of the predictive model, matching the clusters of the points to the segments, and using the matched clusters to refine the positions of the matched segments. It also relates to distinguishing between alternative motions between two observed locations of a control object in a 3D sensory space by accessing first and second positions of a segment of a predictive model of a control object such that motion between the first position and the second position was at least partially occluded from observation in a 3D sensory space.

Claims

1. A computer implemented method of simplifying updating of a predictive model using clustering observed points, the method including:

observing a set of points in a three-dimensional (3D) sensory space;

determining surface normal directions from the points;

clustering the points by their surface normal directions and adjacency;

accessing a predictive model of a hand;

refining positions of segments of the predictive model;

matching the clusters of the points to the segments; and

using the matched clusters to refine the positions of the matched segments.

2. The method of claim 1, further including:

selecting a reference vector and determining a difference in angle between the surface normal directions from the points and the reference vector; and

using a magnitude of the difference to cluster the points.

3. The method of claim 2, wherein the reference vector is orthogonal to a field of view of camera used to capture the points on an image.

4. The method of claim 2, wherein the reference vector is along a longitudinal axis of the hand.

5. The method of claim 2, wherein the reference vector is along a longitudinal axis of a portion of the hand.

6. The method of claim 1, wherein refining positions of segments of the predictive model further includes calculating an error indication by:

determining whether the points and points on the segments of the predictive model are within a threshold closest distance.

7. The method of claim 1, wherein refining positions of segments of the predictive model further includes calculating an error indication by:

pairing the points in the set with points on axes of the segments of the predictive model, wherein the points in the set lie on vectors that are normal to the axes; and

determining a reduced root mean squared deviation (RMSD) of distances between paired point sets.

8. The method of claim 1, wherein refining positions of segments of the predictive model further includes calculating an error indication by:

pairing the points in the set with points on the segments of the predictive model, wherein normal vectors to the points in the set are parallel to each other; and

determining a reduced root mean squared deviation (RMSD) of distances between bases of the normal vectors.

9. The method of claim 1, wherein refining positions of segments of the predictive model further includes:

determining physical proximity between points in the set based on the matched clusters;

based on the determined physical proximity, identifying co-located segments of the predictive model that change positions together; and

refining positions of segments of the predictive model responsive to the co-located segments.

10. The method of claim 9, wherein the co-located segments represent adjoining figures of the hand.

11. The method of claim 9, wherein the co-located segments represent subcomponents of a same finger.

12. A computer implemented method of distinguishing between alternative motions between two observed locations of a control object in a three-dimensional (3D) sensory space, the method including:

accessing first and second positions of a segment of a predictive model of a control object, wherein motion between the first position and the second position was at least partially occluded from observation in a three-dimensional (3D) sensory space;

receiving two or more alternative interpretations of movement from the first position to the second position;

estimating entropy or extent of motion involved in the alternative interpretations;

selecting an alternative interpretation with lower entropy or extent of motion than other interpretations; and

applying the selected interpretation to predicting further positioning of the segment and of other segments of the predictive model from additional observations in the 3D sensory space.

13. The method of claim 12, wherein the control object is a hand.

14. The method of claim 12, wherein the control object is a tool.

15. A system enabling simplifying updating of a predictive model using clustering observed points, comprising:

at least one camera oriented towards a field of view;

a gesture database comprising a series of electronically stored records, each of the records relating a predictive model of a hand; and

an image analyzer coupled to the camera and the database and configured to;

observe a set of points in a three-dimensional (3D) sensory space using at least one image captured by the camera;

determine surface normal directions from the points;

cluster the points by their surface normal directions and adjacency;

access a particular predictive model of the hand;

refine positions of segments of the particular predictive model;

match the clusters of the points to the segments; and

use the matched clusters to refine the positions of the matched segments.

16. The system of claim 15, further configured to:

select a reference vector and determine a difference in angle between the surface normal directions from the points and the reference vector; and

use a magnitude of the difference to cluster the points.

17. The system of claim 16, wherein the reference vector is orthogonal to the field of view.

18. The system of claim 16, wherein the reference vector is along a longitudinal axis of the hand.

19. The system of claim 16, wherein the reference vector is along a longitudinal axis of a portion of the hand.

20. A system to distinguish between alternative motions between two observed locations of a control object in a three-dimensional (3D) sensory space, comprising:

at least one camera oriented towards a field of view;

a gesture database comprising a series of electronically stored records, each of the records relating a predictive model of a hand; and

an image analyzer coupled to the camera and the database and configured to;

access first and second positions of a segment of a predictive model of a control object, wherein motion between the first position and the second position was at least partially occluded from observation in a three-dimensional (3D) sensory space;

receive two or more alternative interpretations of movement from the first position to the second position;

estimate entropy or extent of motion involved in the alternative interpretations;

select an alternative interpretation with lower entropy or extent of motion than other interpretations; and

apply the selected interpretation to predicting further positioning of the segment and of other segments of the predictive model from additional observations in the 3D sensory space.