Dynamic user interactions for display control

• US 9,766,709 B2
• Filed: 03/14/2014
• Issued: 09/19/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David
• Primary Examiner(s): Bhatia, Ajay
• Assistant Examiner(s): Blaufeld, Justin R
• Application Number:
  US14/214,336
• Publication Number:
  US 20140282282A1
• Time in Patent Office: 1,285 Days
• CPC Class Codes: G06F 13/105 where the programme performs an input/output emulation function
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 9/4411 Configuring for operating with peripheral devices; Loading of device drivers

Abstract

The technology disclosed relates to using gestures to supplant or augment use of a standard input device coupled to a system. It also relates to controlling a display using gestures. It further relates to controlling a system using more than one input device. In particular, it relates to detecting a standard input device that causes on-screen actions on a display in response to control manipulations performed using the standard input device. Further, a library of analogous gestures is identified, which includes gestures that are analogous to the control manipulations and also cause the on-screen actions responsive to the control manipulations. Thus, when a gesture from the library of analogous gestures is detected, a signal is generated that mimics a standard signal from the standard input device and causes at least one on-screen action.

Claims

1. A method performed by a gesture recognition and control system for using gestures to supplant or augment use of a standard input device coupled to an electronic device, the method including:

capturing, in a three-dimensional (3D) sensor space, an image including the standard input device and a body part of a user;

identifying the standard input device and the body part of the user included in the captured image;

matching the standard input device in the captured image to an entry of an object template or image in a database, wherein the database maps standard input devices to object templates or images, and wherein each standard input device respectively generates one or more standard signals in response to performing one or more specific control manipulations with each respective standard input device;

detecting that the captured image includes a depiction of a coupling between the matched standard input device and the electronic device;

receiving, after the steps of identifying the body part of the user and detecting that the captured image depicts the matched standard input device coupled to the electronic device, an input from the user indicating the user prefers to control the electronic device with 3D gestures over controlling the electronic device by performing control manipulations directly from the standard input device;

identifying a library of analogous 3D gestures that are analogous to the control manipulations performed using the matched standard input device;

detecting a gesture in the 3D sensor space;

identifying the gesture from the library of analogous 3D gestures;

identifying a signal corresponding to the identified gesture in the library of analogous 3D gestures, the identified signal mimicking the standard signal for performing the specific control manipulation from the matched standard input device; and

providing the identified signal, which mimics the standard signal, to the electronic device instructing the electronic device to perform the specific control manipulation.

2. The method of claim 1, wherein the matched standard input device is a mouse and the control manipulations include at least a right-click, a left-double-click, and a left-click.

3. The method of claim 1, further including populating the library of analogous 3D gestures, including:

receiving a selection from a user of a control manipulation performed using the matched standard input device;

prompting the user to perform a gesture;

assigning the gesture as analogous to the selected control manipulation; and

updating the library of analogous 3D gestures to include the assigned gesture.

4. The method of claim 1, wherein a performance of identified gestures from the library of analogous 3D gestures causes on-screen responsiveness produced by respective control manipulations performed using the matched standard input device.

5. The method of claim 4, wherein the performance of identified gestures from the library of analogous 3D gestures causes an interaction with one or more screen icons responsive to respective control manipulations performed using the matched standard input device.

6. The method of claim 5, wherein the performance of identified gestures from the library of analogous 3D gestures includes gesturing with at least one control object, including:

generating a representation of the control object and of a movement of the control object.

7. The method of claim 6, wherein the generated representation of the control object is a scaled replica of the control object.

8. The method of claim 6, wherein the generated representation of the control object is an animation of the control object.

9. The method of claim 1,

wherein the method further includes, responsive to matching the standard input device and detecting a control object used to perform a gesture from the library of analogous 3D gestures, determining a control paradigm to control on-screen responsiveness based on an active application running on the electronic device, and

wherein the control paradigm identifies whether the active application is responsive to the matched standard input device or the detected control object.

10. The method of claim 1, further including creating the library of analogous 3D gestures analogous to on-screen actions, including:

receiving a selection from a user of an on-screen action;

prompting the user to perform a gesture;

assigning the performed gesture as analogous to the selected on-screen action; and

updating the library of analogous 3D gestures to include the assigned gesture, such that a performance of the assigned gesture generates the selected on-screen action.

11. A method performed by a gesture recognition and control system for controlling a display using gestures, the method including:

capturing, in a three-dimensional (3D) sensor space, an image including a standard input device that causes on-screen actions on the display and a body part of a user;

identifying the standard input device and the body part of the user included in the captured image;

matching the standard input device in the captured image to an entry of an object template or image in a database, wherein the database maps standard input devices to object templates or images, and wherein each standard input device respectively generates one or more standard signals in response to performing one or more specific control manipulations with each respective standard input device;

detecting that the captured image includes a depiction of a coupling between the matched standard input device and the display;

receiving, after the steps of identifying the body part of the user and detecting that the captured image depicts the matched standard input device coupled to the display, an input from the user indicating the user prefers to control the display with 3D gestures over controlling the display by performing control manipulations directly from the standard input device;

identifying a library of analogous 3D gestures that (i) are analogous to the control manipulations performed using the matched standard input device and (ii) cause the on-screen actions responsive to the control manipulations;

detecting a gesture in the 3D sensor space;

identifying the gesture from the library of analogous 3D gestures;

identifying a signal corresponding to the identified gesture in the library of analogous 3D gestures, the identified signal (i) mimicking the standard signal for performing the specific control manipulation from the matched standard input device and (ii) causing the on-screen actions; and

providing the identified signal, which mimics the standard signal, instructing the display to display the on-screen actions.

12. The method of claim 11, wherein the matched standard input device is a mouse and the control manipulations include at least a right-click, a left-double-click, and a left-click.

13. The method of claim 11, further including populating the library of analogous 3D gestures, including:

receiving a selection from a user of a control manipulation performed using the matched standard input device;

prompting the user to perform a gesture;

assigning the gesture as analogous to the selected control manipulation; and

updating the library of analogous 3D gestures to include the assigned gesture.

14. A computer system for using gestures to supplant or augment use of a standard input device coupled to an electronic device, the computer system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

capture, in a three-dimensional (3D) sensor space, an image including the standard input device and a body part of a user;

identify the standard input device and the body part of the user included in the captured image;

match the standard input device in the captured image to an entry of an object template or image in a database, wherein the database maps standard input devices to object templates or images, and wherein each standard input device respectively generates one or more standard signals in response to performing one or more specific control manipulations with each respective standard input device;

detect that the captured image includes a depiction of a coupling between the matched standard input device and the electronic device;

receive, after the identification of the body part of the user and detection that the captured image depicts the matched standard input device coupled to the electronic device, an input from the user indicating the user prefers to control the electronic device with 3D gestures over controlling the electronic device by performing control manipulations directly from the standard input device;

identify a library of analogous 3D gestures that are analogous to the control manipulations performed using the matched standard input device;

detect a gesture in the 3D sensor space;

identify the gesture from the library of analogous 3D gestures;

identify a signal corresponding to the identified gesture in the library of analogous 3D gestures, the identified signal mimicking the standard signal for performing the specific control manipulation from the matched standard input device; and

provide the identified signal, which mimics the standard signal, to the electronic device instructing the electronic device to perform the specific control manipulation.

15. The computer system of claim 14, wherein the computer instructions are further configured to cause the processor to populate the library of analogous 3D gestures, including:

receiving a selection from a user of a control manipulation performed using the matched standard input device;

prompting the user to perform a gesture;

assigning the gesture as analogous to the selected control manipulation; and

updating the library of analogous 3D gestures to include the assigned gesture.

16. The computer system of claim 14, wherein a performance of identified gestures from the library of analogous 3D gestures causes on-screen responsiveness produced by respective control manipulations performed using the matched standard input device.

17. The computer system of claim 14,

wherein the computer instructions are further configured to cause the processor to determine a control paradigm to control on-screen responsiveness based on an active application running on the computer system in response to matching the standard input device and detecting a control object used to perform a gesture from the library of analogous 3D gestures, and

wherein the control paradigm identifies whether the active application is responsive to the matched standard input device or the detected control object.

18. The computer system of claim 14, wherein the computer instructions are further configured to cause the processor to create the library of analogous 3D gestures analogous to on-screen actions, including:

receiving a selection from a user of an on-screen action;

prompting the user to perform a gesture;

assigning the performed gesture as analogous to the selected on-screen action; and

updating the library of analogous 3D gestures to include the assigned gesture, such that a performance of the assigned gesture generates the selected on-screen action.