Dynamic user interactions for display control and identifying dominant gestures

• US 10,564,799 B2
• Filed: 06/29/2017
• Issued: 02/18/2020
• Est. Priority Date: 01/15/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Holz, David S.
• Primary Examiner(s): Nguyen, Tuan S
• Application Number:
  US15/638,205
• Publication Number:
  US 20170300209A1
• Time in Patent Office: 964 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed relates to distinguishing meaningful gestures from proximate non-meaningful gestures in a three-dimensional (3D) sensory space. In particular, it relates to calculating spatial trajectories of different gestures and determining a dominant gesture based on magnitudes of the spatial trajectories. The technology disclosed also relates to uniformly responding to gestural inputs from a user irrespective of a position of the user. In particular, it relates to automatically adapting a responsiveness scale between gestures in a physical space and resulting responses in a gestural interface by automatically proportioning on-screen responsiveness to scaled movement distances of gestures in the physical space, user spacing with the 3D sensory space, or virtual object density in the gestural interface. The technology disclosed further relates to detecting if a user has intended to interact with a virtual object based on measuring a degree of completion of gestures and creating interface elements in the 3D space.

Claims

1. A method of creating interface elements in a three-dimensional (3D) sensory space, the method including:

detecting a circular sweep made by a finger of a hand detected to remain stationary during the circular sweep made by the finger in a three-dimensional (3D) sensory space using an electronic sensor;

constructing an on-screen button in a gestural interface responsive to the circular sweep;

detecting subsequently a lateral sweep of the finger in the 3D sensory space; and

registering a pressing of the on-screen button responsive to the lateral sweep and performing at least one associated function;

wherein the associated function is selected based on a location of the on-screen button on the gestural interface.

2. The method of claim 1, wherein the associated function is selected based on a context of the gestural interface.

3. The method of claim 2, the context of the gestural interface further including determining that a user in proximity to a television performs a first gesture detected by the electronic sensor;

and in response to the first gesture, issuing a command to the television to activate and display a control panel on screen, the control panel allowing the user to make selections thereon using subsequent gestures.

4. The method of claim 3, further including determining using the electronic sensor that the user has after making the first gesture performed various gestures to remotely interact with virtual objects in a virtual environment defined by a video game;

detecting a subsequent hand gesture made by the user; and

providing the gestures detected as an input parameter to a currently running game that interprets the gestures and takes at least one context-appropriate action.

5. The method of claim 4, further including detecting using the electronic sensor a gesture made by the user;

and responsively manipulate data displayed on the control panel to perform at least one selected from a set consisting of selecting a channel of interest, adjusting audio sound, and varying a screen brightness of a screen.

6. The method of claim 5, wherein a degree of completion is determined for the gesture made by the user by matching the gesture as detected by the electronic sensor to a gesture in a database that includes multiple images or vectors, wherein each image or vector is associated with a degree of completion of the gesture;

and the method further including rendering the degree of completion of the performed gesture on the television.

7. The method of claim 5, wherein a degree of completion is determined for the gesture made by the user is computed by an interpolation or a comparison of the observed gesture to a stored vector;

and the method further including rendering the degree of completion of the performed gesture on the television.

8. The method of claim 1, further including interpreting the lateral sweep as a left mouse click when at least a threshold percentage of its motion is in a direction the finger is pointing.

9. The method of claim 1, further including interpreting the lateral sweep as a right mouse click when at least a threshold percentage of its motion is in a direction opposite to which the finger is pointing.

10. The method of claim 1, further including dynamically controlling a relationship between actual movement of the user and a corresponding action displayed on-screen by:

analyzing at least some motion of the finger present in a series of captured images and producing a single rotation factor based on the at least some motion as analyzed;

using the single rotation factor produced as an input to a program running on a system; and

rendering reconstructed 3D images of the gestures, thereby enabling dynamic user interactions for display control in real time.

11. The method of claim 1, further including responsive to the lateral sweep issuing a command to control a machine coupled with the electronic sensor.

12. The method of claim 1, further including responsive to the lateral sweep issuing a command to the electronic sensor.

13. The method of claim 12, further including the command issued controlling a component of the electronic sensor.

14. The method of claim 1, further including receiving input from a user indicating a size of a physical movement of the user;

and adjusting a relationship between the physical movement of the user and corresponding actions displayed on a screen.

15. The method of claim 1, further including detecting an additional movement of at least one other finger of the hand made during the circular sweep made by the finger in a three-dimensional (3D) sensory space using an electronic sensor;

determining magnitudes of spatial trajectories at least in part by a scale of curling of the fingers and degrees of freedom between the fingers; and

evaluating magnitudes of respective spatial trajectories and determining a dominant gesture based on magnitudes of the spatial trajectories.

16. The method of claim 15, further including triggering a response to the dominant gesture determined without triggering a response to a non-dominant gesture.

17. A system including a processor and a computer readable storage unit storing instructions that when executed by the processor perform:

detecting a circular sweep made by a finger of a hand detected to remain stationary during the circular sweep made by the finger in a three-dimensional (3D) sensory space using an electronic sensor;

constructing an on-screen button in a gestural interface responsive to the circular sweep;

detecting subsequently a lateral sweep of the finger in the 3D sensory space; and

registering a pressing of the on-screen button responsive to the lateral sweep and performing at least one associated function;

wherein the associated function is selected based on a location of the on-screen button on the gestural interface.

18. A non-transitory computer readable medium storing instructions that when executed by a processor perform:

detecting a circular sweep made by a finger of a hand detected to remain stationary during the circular sweep made by the finger in a three-dimensional (3D) sensory space using an electronic sensor;

constructing an on-screen button in a gestural interface responsive to the circular sweep;

detecting subsequently a lateral sweep of the finger in the 3D sensory space; and

registering a pressing of the on-screen button responsive to the lateral sweep and performing at least one associated function;

wherein the associated function is selected based on a location of the on-screen button on the gestural interface.