Non-linear motion capture using Frenet-Serret frames

• US 9,857,876 B2
• Filed: 07/22/2014
• Issued: 01/02/2018
• Est. Priority Date: 07/22/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Hare, Gabriel A, Mertens, Keith, Perez, Matias, Roy, Neeloy, Holz, David
• Primary Examiner(s): HICKS, CHARLES V
• Application Number:
  US14/338,136
• Publication Number:
  US 20150022447A1
• Time in Patent Office: 1,260 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Implementations of the technology disclosed convert captured motion from Cartesian/(x,y,z) space to Frenet-Serret frame space, apply one or more filters to the motion in Frenet-Serret space, and output data (for display or control) in a desired coordinate space—e.g., in a Cartesian/(x,y,z) reference frame. The output data can better represent a user'"'"'s actual motion or intended motion.

Claims

1. A method of accurately capturing a non-linear gestural path of a hand-gesture in three-dimensional (3D) sensory space, the method including:

capturing with a camera a sequence of multiple images of a user'"'"'s hand making a non-linear free-form hand-gesture performed by the user'"'"'s hand, moving freely and substantially independently of contact with other objects in a three-dimensional (3D) sensory space monitored by the camera;

using a first coordinate system associated with the camera, determining from the sequence of multiple images a first set of coordinates of a plurality of 3D positions along a hand-gesture movement path made by the user'"'"'s hand during the non-linear free-form hand-gesture captured;

determining an orientation invariant trajectory for the hand-gesture movement path during the non-linear free-form hand-gesture by transforming the first set of coordinates of the plurality of 3D positions along the hand-gesture movement path captured in the first coordinate system into a second set of coordinates in a second coordinate system according to a transform defining a relationship between the first coordinate system and the second coordinate system, including;

transforming points in the images captured representing positions in space of portions of the user'"'"'s hand captured using the first coordinate system corresponding to coordinates of the camera into corresponding points using the second coordinate system by attaching to select ones of the points a frame of reference consisting of an orthonormal vector set and

building a sequence of orthonormal vector sets at sequential points of the select ones of the points thereby describing motion of the user'"'"'s hand at the select ones of the points as the sequence of orthonormal vector sets;

applying at least one filter to the second set of coordinates to obtain filtered curvilinear motion data for the hand-gesture movement path;

comparing the orientation invariant trajectory for the hand-gesture movement path to a library of known gestures electronically stored as records in a database to detect a gesture; and

when a similar trajectory for a user'"'"'s hand located in the library of known gesture is found, providing a command input to control a system based upon the gesture detected; and

generating for display a smoothened representation of the hand-gesture movement path using an approximate best-fit curve that predicts a trajectory for the user'"'"'s hand by connecting points along the hand-gesture movement path.

2. The method of claim 1, further including:

recognizing a second hand-gesture as corresponding to a recording of the hand-gesture movement path by;

determining an orientation invariant trajectory of a second hand-gesture movement path of the second hand-gesture; and

matching the orientation invariant trajectory of the second hand-gesture movement path to the orientation invariant trajectory of the hand-gesture movement path;

thereby eliminating from consideration translation or rotation of the second hand-gesture relative to the recording of the hand-gesture movement path.

3. The method of claim 1, further including using the command input to control and interact with computer systems by moving fingers in air.

4. The method of claim 1, further including using the command input to perform free-floating handwriting recognition.

5. The method of claim 4, the free-floating handwriting recognition further including applying the smoothed representation to disambiguate between similarly shaped trajectories of handwriting characters in the 3D sensory space.

6. The method of claim 1, further including the hand holding a stylus.

7. The method of claim 1, wherein the orthonormal vector set includes:

a tangent unit vector that is tangent to the hand-gesture movement path and pointing in a direction of motion of the user'"'"'s hand at a corresponding point, a normal unit vector that is derivative of the tangent unit vector with respect to an arclength of at least a portion of the hand-gesture movement path divided by a length of the at least a portion of the hand-gesture movement path, and a binormal unit vector comprising a cross product of the tangent unit vector and the normal unit vector.

8. The method of claim 1, further including associating a frame of reference with at least two locations along the hand-gesture movement path and having no corresponding point in the images captured.

9. The method of claim 8, further including distributing the at least two locations evenly along the hand-gesture movement path.

10. The method of claim 8, further including distributing additional frames of reference at locations along the hand-gesture movement path in which the hand-gesture movement path changes direction more rapidly than at any other points in the hand-gesture movement path.

11. The method of claim 1, further including:

determining a curvature and a torsion of the hand-gesture movement path from a rotation between two consecutive ones of the sequence of orthonormal vector sets for the select ones of the points along the hand-gesture movement path; and

removing from the hand-gesture movement path, points indicated by the curvature and torsion to correspond to unintended motion;

thereby creating a resultant hand-gesture movement path.

12. The method of claim 11, further including:

replacing points removed from the hand-gesture movement path with a curve defined by Euler'"'"'s spiral fitted to the hand-gesture movement path in the second coordinate system; and

transforming the hand-gesture movement path back to the first coordinate system.

13. The method of claim 12, further including:

fitting a spiral having a curvature near zero at a tangent portion and increasing linearly with its curve length such that when the spiral meets a circular portion of the hand-gesture movement path, curvature of the spiral is equal to curvature of the hand-gesture movement path.

14. The method of claim 11, further including:

using the resultant hand-gesture movement path, predicting an intended free-form hand-gesture for a user before the non-linear free-form hand-gesture has been completed.

15. The method of claim 1, further including constructing for display an Euler spiral to model boundary segments between at least a first extreme and a second extreme of the hand-gesture movement path.

16. A non-transitory computer readable medium storing instructions to accurately capture a non-linear gestural path of a hand-gesture in three-dimensional (3D) sensory space, which instructions when executed by a processor perform actions including:

capturing with a camera a sequence of multiple images of a user'"'"'s hand making a non-linear free-form hand-gesture performed by the user'"'"'s hand, moving freely and substantially independently of contact with other objects in a three-dimensional (3D) sensory space monitored by the camera;

using a first coordinate system associated with the camera, determining from the sequence of multiple images a first set of coordinates of a plurality of 3D positions along a hand-gesture movement path made by the user'"'"'s hand during the non-linear free-form hand-gesture captured;

determining an orientation invariant trajectory for the hand-gesture movement path during the non-linear free-form hand-gesture by transforming the first set of coordinates of the plurality of 3D positions along the hand-gesture movement path captured in the first coordinate system into a second set of coordinates in a second coordinate system according to a transform defining a relationship between the first coordinate system and the second coordinate system, including;

transforming points in the images captured representing positions in space of portions of the user'"'"'s hand captured using the first coordinate system corresponding to coordinates of the camera into corresponding points using the second coordinate system by attaching to select ones of the points a frame of reference consisting of an orthonormal vector set and

building a sequence of orthonormal vector sets at sequential points of the select ones of the points thereby describing motion of the user'"'"'s hand at the select ones of the points as the sequence of orthonormal vector sets;

applying at least one filter to the second set of coordinates to obtain filtered curvilinear motion data for the hand-gesture movement path; and

comparing the orientation invariant trajectory for the hand-gesture movement path to a library of known gestures electronically stored as records in a database to detect a gesture; and

when a similar trajectory for a user'"'"'s hand located in the library of known gesture is found, providing a command input to control a system based upon the gesture detected; and

generating for display a smoothened representation of the hand-gesture movement path using an approximate best-fit curve that predicts a trajectory for the user'"'"'s hand by connecting points along the hand-gesture movement path.

17. A system of accurately capturing non-linear gestural path of a hand-gesture in three-dimensional (3D) sensory space, the system including:

at least one camera to capture a sequence of multiple images of a user'"'"'s hand making a non-linear free-form gesture performed by the user'"'"'s hand, moving freely and substantially independently of contact with other objects in a three-dimensional (3D) sensory space monitored by the camera;

a processor;

a processor-executable object-detection model to determine from the sequence of multiple images a first set of coordinates of a plurality of 3D positions along a hand-gesture movement path made by the user'"'"'s hand during a non-linear free-form hand-gesture captured using a first coordinate system associated with the camera;

a processor-executable filtering module to;

determine an orientation invariant trajectory for the hand-gesture movement path by transforming the first set of coordinates of the plurality of 3D positions along the hand-gesture movement path captured in the first coordinate system into a second set of coordinates in a second coordinate system according to a transform defining a relationship between the first coordinate system and the second coordinate system, including;

transforming points in the images captured representing positions in space of portions of the user'"'"'s hand captured using the first coordinate system corresponding to coordinates of the camera into corresponding points using the second coordinate system by attaching to select ones of the points a frame of reference consisting of an orthonormal vector set and

building a sequence of orthonormal vector sets at sequential points of the select ones of the points thereby describing motion of the user'"'"'s hand at the select ones of the points as the sequence of orthonormal vector sets;

apply at least one filter to the second set of coordinates to obtain filtered curvilinear motion data for the hand-gesture movement path; and

compare the orientation invariant trajectory for the hand-gesture movement path to a library of known gestures electronically stored as records in a database to detect a gesture; and

when a similar trajectory for a user'"'"'s hand located in the library of known gesture is found, providing a command input to control a system based upon the gesture detected; and

generate for display a smoothened representation of the hand-gesture movement path using an approximate best-fit curve that predicts a trajectory for the user'"'"'s hand by connecting points along the hand-gesture movement path.