Predictive information for free space gesture control and communication

• US 9,934,609 B2
• Filed: 07/31/2017
• Issued: 04/03/2018
• Est. Priority Date: 08/29/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Horowitz, Kevin A., Holz, David S.
• Primary Examiner(s): Mazumder, Tapas
• Application Number:
  US15/664,959
• Publication Number:
  US 20170330374A1
• Time in Patent Office: 246 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20048 Transform domain processing
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Free space machine interface and control can be facilitated by predictive entities useful in interpreting a control object'"'"'s position and/or motion (including objects having one or more articulating members, i.e., humans and/or animals and/or machines). Predictive entities can be driven using motion information captured using image information or the equivalents. Predictive information can be improved applying techniques for correlating with information from observations.

Claims

1. A method of capturing gestural motion of a control object in a three-dimensional (3D) sensory space, the method including:

determining observation information characterizing a surface of a control object from at least one image of a gestural motion of the control object in a three-dimensional (3D) sensory space;

constructing a 3D solid model to represent the control object by fitting one or more 3D solid subcomponents to the surface characterized; and

improving representation of the gestural motion by the 3D solid model, including;

pairing point sets on the surface with points on axes of the 3D solid subcomponents, wherein surface points lie on vectors that are normal to the axes;

determining an error indication between a point on the surface characterized of the control object and a corresponding point on at least one of the 3D solid subcomponents; and

responsive to the error indication adjusting the 3D solid model.

2. The method of claim 1, wherein determining the error indication further includes determining whether the point on the surface and the corresponding point on the at least one of the 3D solid subcomponents are within a threshold distance.

3. The method of claim 1, wherein determining the error indication further includes:

determining a reduced root mean squared deviation (RMSD) of distances between paired point sets.

4. The method of claim 1, wherein determining the error indication further includes:

pairing point sets on the surface with points on the 3D solid subcomponents, wherein normal vectors to the points sets are parallel to each other; and

determining a reduced root mean squared deviation (RMSD) of distances between bases of the normal vectors.

5. The method of claim 1, wherein fitting the one or more 3D solid subcomponents further includes fitting a set of closed curves to at least a portion of the surface.

6. The method of claim 5, wherein closed curves include at least one of radial solids, capsuloids, spheres, ellipsoids, and hyperboloids.

7. The method of claim 5, wherein at least one closed curve includes a contour and a surface defined by a set of points at a radius from the contour.

8. The method of claim 1, wherein the control object is a hand and fitting the one or more 3D solid subcomponents further includes at least one of:

fitting capsuloids in finger portions of the surface; and

fitting radial solids in palm and/or wrist portions of the surface.

9. The method of claim 1, wherein adjusting the 3D solid model further includes altering the 3D solid subcomponents to conform to at least one of length, width, orientation, and arrangement of portions of the surface.

10. The method of claim 9, wherein altering the 3D solid subcomponents further includes applying a transformation matrix to a plurality of points on the 3D solid subcomponents.

11. The method of claim 9, wherein altering the 3D solid subcomponents further includes determining a rotation matrix that provides a reduced root mean squared deviation (RMSD) between paired point sets on the surface and point sets on the 3D solid subcomponents.

12. The method of claim 1, further including determining the observation information by fitting a family of closed curves and positions of a plurality of two-dimensional (2D) cross-sectional portions of the control object to the image based at least on a location of at least one image capturing device.

13. The method of claim 1, further including improving representation of the gestural motion by the 3D solid model by interpolating 3D solid model positions across time based on expected continuity in motion and deformation of the control object.

14. The method of claim 1, wherein constructing the 3D solid model further includes selecting a pre-determined 3D solid model from an object library based on characteristic parameters of the control object.

15. The method of claim 1, further including improving representation of the gestural motion by the 3D solid model by:

detecting fits of 3D solid subcomponents with colliding subcomponents; and

fitting, to the surface, 3D solid subcomponents with least colliding subcomponents.

16. The method of claim 15, wherein detection of colliding subcomponents is based at least on identifying a subcomponent attribute incompatible with an adjacent subcomponent attribute.

17. The method of claim 1, wherein constructing the 3D solid model further includes determining the 3D solid subcomponents from physical characteristics of a type of control object being observed.

18. The method of claim 17, wherein the control object is a hand and the physical characteristics of the hand include at least one of:

four fingers and a thumb of the hand;

a palm to which the fingers and the thumb are connected; and

positions and angles of the fingers and the thumb relative to each other and to the palm.

19. The method of claim 17, wherein the control object is a tool and the physical characteristics of the tool include at least one of:

length of the tool;

width of the tool; and

pointing direction vector of the tool.

20. The method of claim 1, further including:

identifying one or more 3D solid subcomponents at an extremity of the control object; and

constraining error indication determination to subcomponents identified at an extremity.

21. The method of claim 1, further including:

linking the 3D solid subcomponents by representing a plurality of subcomponents using one or more artificial constructs; and

constraining error indication determination to the artificial constructs.

22. The method of claim 21, wherein the 3D solid subcomponents are fitted to a hand surface, further including at least one of:

using a line segment to represent finger subcomponents; and

using at least one of a line, square, circle, and ellipse to represent palm subcomponents.

23. The method of claim 1, further including:

defining a surface of a control object using a web of artificial constructs;

identifying surface portions of the control object using separate clusters of artificial constructs; and

constraining error indication determination to artificial constructs or clusters thereof.

24. The method of claim 1, further including improving representation of the gestural motion by the 3D solid model by:

detecting conflicting attributes between adjacent 3D solid subcomponents; and

fitting, to the surface, 3D solid subcomponents with least conflicting attributes.

25. The method of claim 24, further including:

ranking the 3D solid subcomponents with conflicting attributes based on a degree of conflict; and

presenting ranked 3D solid subcomponents for selection.

26. The method of claim 24, wherein conflicting attributes include at least one of minima and maxima of intersection angles between the 3D solid subcomponents.

27. The method of claim 1, further including:

repeatedly determining observation information and improving the 3D solid model constructed using the observation information; and

determining gestural motion of the control object based on differences between 3D solid model positions across time.

28. A non-transitory computer readable storage medium impressed with computer program instructions to capture gestural motion of a control object in a three-dimensional (3D) sensory space, which instructions, when executed on a processor, implement a method including:

determining observation information characterizing a surface of a control object from at least one image of a gestural motion of the control object in a three-dimensional (3D) sensory space;

constructing a 3D solid model to represent the control object by fitting one or more 3D solid subcomponents to the surface characterized; and

improving representation of the gestural motion by the 3D solid model, including;

pairing point sets on the surface with points on axes of the 3D solid subcomponents, wherein surface points lie on vectors that are normal to the axes;

determining an error indication between a point on the surface characterized of the control object and a corresponding point on at least one of the 3D solid subcomponents; and

responsive to the error indication adjusting the 3D solid model.

29. A system for capturing gestural motion of a control object in a three-dimensional (3D) sensory space, comprising:

a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to;

determine observation information characterizing a surface of a control object from at least one image of a gestural motion of the control object in a three-dimensional (3D) sensory space;

construct a 3D solid model to represent the control object by fitting one or more 3D solid subcomponents to the surface characterized; and

improve representation of the gestural motion by the 3D solid model, including;

pairing point sets on the surface with points on axes of the 3D solid subcomponents, wherein surface points lie on vectors that are normal to the axes;

determining an error indication between a point on the surface characterized of the control object and a corresponding point on at least one of the 3D solid subcomponents; and

responsive to the error indication adjusting the 3D solid model.