Dynamic user interactions for display control and measuring degree of completeness of user gestures

• US 10,042,510 B2
• Filed: 01/15/2014
• Issued: 08/07/2018
• Est. Priority Date: 01/15/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David
• Primary Examiner(s): Nguyen, Tuan S
• Application Number:
  US14/156,424
• Publication Number:
  US 20140201683A1
• Time in Patent Office: 1,665 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed relates to distinguishing meaningful gestures from proximate non-meaningful gestures in a three-dimensional (3D) sensory space. In particular, it relates to calculating spatial trajectories of different gestures and determining a dominant gesture based on magnitudes of the spatial trajectories. The technology disclosed also relates to uniformly responding to gestural inputs from a user irrespective of a position of the user. In particular, it relates to automatically adapting a responsiveness scale between gestures in a physical space and resulting responses in a gestural interface by automatically proportioning on-screen responsiveness to scaled movement distances of gestures in the physical space, user spacing with the 3D sensory space, or virtual object density in the gestural interface. The technology disclosed further relates to detecting if a user has intended to interact with a virtual object based on measuring a degree of completion of gestures and creating interface elements in the 3D space.

Claims

1. A method of creating interface elements in a three-dimensional (3D) sensory space, the method including:

detecting a first motion made by a finger in a three-dimensional (3D) sensory space using an electronic sensor including capturing by the cameras images of the first motion made by the finger and analyzing the images to detect edges of objects including the finger and other fingers;

determining that the first motion by the finger includes a first swirl gesture by and representing gesture-relevant body parts including the finger and other fingers using a simplified skeletal model and determining therefrom gestures made by individual fingers, wherein;

the finger making the first motion is a finger that is determined to be moving in a swirling motion; and

the finger is attached to a hand that is determined to not move along with the finger that is determined to be moving in the swirling motion;

determining, based on magnitudes of their respective spatial trajectories, whether to recognize a gesture associated with (a) the first motion of the finger and (b) lack of motion of the hand to which the finger is attached;

recognizing a gesture associated with the first motion of the finger without motion of the hand to which the finger is attached;

constructing an on-screen puck that is constructed having at least one associated function, wherein the on-screen puck is constructed responsive to the first motion made by the finger without motion of the hand to which the finger is attached;

detecting in additional images, captured using the cameras of the electronic sensor, a subsequent second motion of both the finger and an extended second finger in the 3D sensory space, wherein;

the finger making the first motion and the extended second finger are determined to be moving in second motion; and

the detected subsequent second motion being different from the first motion;

determining, based on magnitudes of their respective spatial trajectories, whether to recognize a gesture associated with (a) the second motion of the finger and (b) the extended second finger;

recognizing a gesture associated with the second motion of the finger and the extended second finger that is different from the gesture associated with the first motion of the finger and the extended second finger; and

rotating the on-screen puck responsive to the subsequent second motion of the finger and the extended second finger moving in the 3D sensory space and performing the at least one associated function.

2. The method of claim 1, further including moving a hand portion other than the finger.

3. The method of claim 1, further including detecting a repeatedly performed clockwise or anti-clockwise movement of the finger in free space at least substantially concurrent with an upward or downward movement of the finger.

4. The method of claim 1, further including the on-screen puck continuing to move for a period of time after a user ceases making the subsequent second motion of the finger.

5. The method of claim 1, further including the on-screen puck springing back to a fixed position after a user ceases making the subsequent second motion of the finger.

6. The method of claim 1, wherein the second motion is recognized when the finger traverses a distance of between 1 and 5 millimeters.

7. The method of claim 1, wherein the electronic sensor has cameras to detect the finger moving freely in space.

8. A system including one or more processors coupled to memory storing computer instructions to create interface elements in a three-dimensional (3D) sensory space, which computer instructions, when executed on the processors, implement actions comprising:

detecting a first motion made by a finger in a three-dimensional (3D) sensory space using an electronic sensor including capturing by the cameras images of the first motion made by the finger and analyzing the images to detect edges of objects including the finger and other fingers;

determining that the first motion by the finger includes a first swirl gesture by and representing gesture-relevant body parts including the finger and other fingers using a simplified skeletal model and determining therefrom gestures made by individual fingers, wherein;

the finger making the first motion is a finger that is determined to be moving in a swirling motion; and

the finger is attached to a hand that is determined to not move along with the finger that is determined to be moving in the swirling motion;

determining, based on magnitudes of their respective spatial trajectories, whether to recognize a gesture associated with (a) the first motion of the finger and (b) lack of motion of the hand to which the finger is attached;

recognizing a gesture associated with the first motion of the finger without motion of the hand to which the finger is attached;

constructing an on-screen puck that is constructed having at least one associated function, wherein the on-screen puck is constructed responsive to the first motion made by the finger without motion of the hand to which the finger is attached;

detecting in additional images, captured using the cameras of the electronic sensor, a subsequent second motion of both the finger and an extended second finger in the 3D sensory space, wherein;

the finger making the first motion and the extended second finger are determined to be moving in second motion; and

the detected subsequent second motion being different from the first motion;

determining, based on magnitudes of their respective spatial trajectories, whether to recognize a gesture associated with (a) the second motion of the finger and (b) the extended second finger;

recognizing a gesture associated with the second motion of the finger and the extended second finger that is different from the gesture associated with the first motion of the finger and the extended second finger; and

rotating the on-screen puck responsive to the subsequent second motion of the finger and the extended second finger moving in the 3D sensory space and performing the at least one associated function.

9. The system of claim 8, implementing actions comprising moving a hand portion other than the finger.

10. The system of claim 8, further implementing actions comprising detecting a repeatedly performed clockwise or anti-clockwise movement of the finger in free space at least substantially concurrent with an upward or downward movement of the finger.

11. The system of claim 8, further implementing actions comprising the on-screen puck continuing to move for a period of time after a user ceases making the subsequent second motion of the finger.

12. The system of claim 8, further implementing actions comprising the on-screen puck springing back to a fixed position after a user ceases making the subsequent second motion of the finger.

13. A non-transitory computer readable storage medium impressed with computer program instructions to create interface elements in a three-dimensional (3D) sensory space, which computer program instructions, when executed on one or more processors, implement actions comprising:

detecting a first motion made by a finger in a three-dimensional (3D) sensory space using an electronic sensor including capturing by the cameras images of the first motion made by the finger and analyzing the images to detect edges of objects including the finger and other fingers;

determining that the first motion by the finger includes a first swirl gesture by and representing gesture-relevant body parts including the finger and other fingers using a simplified skeletal model and determining therefrom gestures made by individual fingers, wherein;

the finger making the first motion is a finger that is determined to be moving in a swirling motion; and

the finger is attached to a hand that is determined to not move along with the finger that is determined to be moving in the swirling motion;

determining, based on magnitudes of their respective spatial trajectories, whether to recognize a gesture associated with (a) the first motion of the finger and (b) lack of motion of the hand to which the finger is attached;

recognizing a gesture associated with the first motion of the finger without motion of the hand to which the finger is attached;

constructing an on-screen puck that is constructed having at least one associated function, wherein the on-screen puck is constructed responsive to the first motion made by the finger without motion of the hand to which the finger is attached;

detecting in additional images, captured using the cameras of the electronic sensor, a subsequent second motion of both the finger and an extended second finger in the 3D sensory space, wherein;

the finger making the first motion and the extended second finger are determined to be moving in second motion; and

the detected subsequent second motion being different from the first motion;

determining, based on magnitudes of their respective spatial trajectories, whether to recognize a gesture associated with (a) the second motion of the finger and (b) the extended second finger;

recognizing a gesture associated with the second motion of the finger and the extended second finger that is different from the gesture associated with the first motion of the finger and the extended second finger; and

rotating the on-screen puck responsive to the subsequent second motion of the finger and the extended second finger moving in the 3D sensory space and performing the at least one associated function.

14. The non-transitory computer readable storage medium of claim 13, wherein the implement actions further comprises moving a hand portion other than the finger.

15. The non-transitory computer readable storage medium of claim 13, wherein the implement actions further comprises detecting a repeatedly performed clockwise or anti-clockwise movement of the finger in free space at least substantially concurrent with an upward or downward movement of the finger.

16. The non-transitory computer readable storage medium of claim 13, wherein the implement actions further comprises the on-screen puck continuing to move for a period of time after a user ceases making the subsequent second motion of the finger.

17. The non-transitory computer readable storage medium of claim 13, wherein the implement actions further comprises the on-screen puck springing back to a fixed position after a user ceases making the subsequent second motion of the finger.

18. A method of creating interface elements in a three-dimensional (3D) sensory space, the method comprising:

detecting a first motion made by a finger in the 3D sensory space using an electronic sensor;

detecting, by the electronic sensor, an arm waving, wherein;

the finger making the first motion is a finger on a hand attached to the waving arm;

the arm moves a greater distance through the 3D sensory space than a distance that the finger moves relative to the hand; and

the finger moves a distance of 1 to 5 millimeters relative to the hand;

detecting positions of the arm, the hand attached to the arm, and fingers attached to the hand in the 3D sensory space;

calculating from the detected positions a spatial trajectory of a waving motion executed by the arm and motion made by the finger;

determining, based on magnitudes of the respective spatial trajectories, whether to recognize a gesture associated with (a) the waving motion or (b) the motion of the finger;

recognizing a gesture associated with the first motion of the finger without recognizing a gesture associated with the arm waving motion; and

manipulating an on-screen item, constructed in response to a detected motion, according to the recognized gesture.