DYNAMIC USER INTERACTIONS FOR DISPLAY CONTROL

• US 20140282282A1
• Filed: 03/14/2014
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: HOLZ, David
• Granted Patent: US 9,766,709 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 13/105 where the programme performs an input/output emulation function
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 9/4411 Configuring for operating with peripheral devices; Loading of device drivers

Abstract

The technology disclosed relates to using gestures to supplant or augment use of a standard input device coupled to a system. It also relates to controlling a display using gestures. It further relates to controlling a system using more than one input device. In particular, it relates to detecting a standard input device that causes on-screen actions on a display in response to control manipulations performed using the standard input device. Further, a library of analogous gestures is identified, which includes gestures that are analogous to the control manipulations and also cause the on-screen actions responsive to the control manipulations. Thus, when a gesture from the library of analogous gestures is detected, a signal is generated that mimics a standard signal from the standard input device and causes at least one on-screen action.

Claims

1. A method of using gestures to supplant or augment use of a standard input device coupled to a system, the method including:

detecting a standard input device coupled to a system and identifying a library of analogous gestures that are analogous to control manipulations performed using the standard input device;

detecting a gesture in a 3D sensor space and identifying the gesture from the library of analogous gestures; and

supplying a signal to the system that mimics a standard signal from the standard input device, based on the detected and identified gesture.

2. The method of claim 1, wherein the standard input device generates particular standard signals in response to performance of respective control manipulations, further including supplying the particular standard signals to the system in response to performance of identified gestures from the library of analogous gestures as a substitute for performance of the respective control manipulations.

3. The method of claim 1, wherein the standard input device is a mouse and the control manipulations include at least right-click, left-double-click, and left-click.

4. The method of claim 1, further including populating the library of analogous gestures, including:

receiving a selection from a user of a control manipulation performed using the standard input device;

prompting the user to perform a gesture; and

assigning the gesture as analogous to the selected control manipulation and updating the library of analogous gestures to include the assigned gesture.

5. The method of claim 1, wherein performance of identified gestures from the library of analogous gestures causes on-screen responsiveness produced by respective control manipulations performed using the standard input device.

6. The method of claim 5, wherein performance of identified gestures from the library of analogous gestures causes interaction with one or more screen icons responsive to respective control manipulations performed using the standard input device.

7. The method of claim 6, wherein performance of identified gestures from the library of analogous gestures includes gesturing with at least one control object, further including:

generating a representation of the control object and of movement of the control object.

8. The method of claim 7, wherein the generated representation is a scaled replica of the control object.

9. The method of claim 7, wherein the generated representation is an animation of the control object.

10. The method of claim 1, further including responsive to detecting the standard input device and a control object used to performs gesture from the library of analogous gestures, determining a control paradigm to control on-screen responsiveness based on an active application running on the system, wherein the control paradigm identifies whether the active application is responsive to the standard input device or the control object.

11. The method of claim 1, further including creating a library of analogous gestures analogous to on-screen actions, including:

receiving a selection from a user of an on-screen action;

prompting the user to perform a gesture; and

assigning the performed gesture as analogous to the selected on-screen action; and

updating the library of gestures to include the assigned gesture such that performance of the assigned gesture generates the selected on-screen action.

12. A method of controlling a display using gestures, the method including:

detecting a standard input device that causes on-screen actions on a display in response to control manipulations performed using the standard input device;

identifying a library of analogous gestures that are analogous to the control manipulations and cause the on-screen actions responsive to the control manipulations;

detecting a gesture in a 3D sensor space and identifying the gesture from the library of analogous gestures; and

generating a signal that mimics a standard signal from the standard input device and causing at least one on-screen action, based on the detected and identified gesture.

13. The method of claim 12, wherein the standard input device is a mouse and the control manipulations include at least right-click, left-double-click, and left-click.

14. The method of claim 12, further including populating the library of analogous gestures, including:

receiving a selection from a user of a control manipulation performed using the standard input device;

prompting the user to perform a gesture; and

assigning the gesture as analogous to the selected control manipulation and updating the library of analogous gestures to include the assigned gesture.

15. A computer system of using gestures to supplant or augment use of a standard input device coupled to a system, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

detect a standard input device coupled to a system and identifying a library of analogous gestures that are analogous to control manipulations performed using the standard input device;

detect a gesture in a 3D sensor space and identifying the gesture from the library of analogous gestures; and

supply a signal to the system that mimics a standard signal from the standard input device, based on the detected and identified gesture.

16. The computer system of claim 15, wherein the standard input device generates particular standard signals in response to performance of respective control manipulations, further configured to supply the particular standard signals to the system in response to performance of identified gestures from the library of analogous gestures as a substitute for performance of the respective control manipulations.

17. The computer system of claim 15, further configured to populate the library of analogous gestures, including:

receiving a selection from a user of a control manipulation performed using the standard input device;

prompting the user to perform a gesture; and

assigning the gesture as analogous to the selected control manipulation and updating the library of analogous gestures to include the assigned gesture.

18. The computer system of claim 15, wherein performance of identified gestures from the library of analogous gestures causes on-screen responsiveness produced by respective control manipulations performed using the standard input device.

19. The computer system of claim 15, further configured to determine a control paradigm to control on-screen responsiveness based on an active application running on the system in response to detecting the standard input device and a control object used to performs gesture from the library of analogous gestures, wherein the control paradigm identifies whether the active application is responsive to the standard input device or the control object.

20. The computer system of claim 15, further configured to create a library of analogous gestures analogous to on-screen actions, including:

receiving a selection from a user of an on-screen action;

prompting the user to perform a gesture; and

assigning the performed gesture as analogous to the selected on-screen action; and

updating the library of gestures to include the assigned gesture such that performance of the assigned gesture generates the selected on-screen action.

21. A method of controlling a system using more than one input device, the method including:

detecting a presence of a first input device coupled to a system and determining a first input device type therefrom;

detecting with a second input device, of a second device type, at least one gesture made by a hand in a 3D sensor space adjacent to, but without required contact with, the second input device and indicating an input to the system;

identifying in a library based upon the at least one gesture, at least one control manipulation which when applied to the first input device, would provide command information analogous to the input to the system indicated by the at least one gesture; and

supplying a signal to the system that emulates the control manipulation applied to the first input device.

22. The method of claim 21, wherein the detecting a presence of a first device includes detecting a mouse coupled to the system and wherein the command information includes a mouse button click.

23. The method of claim 21, wherein detecting with a second device includes detecting a gesture using an imaging input sensor.

24. The method of claim 21, wherein the identifying in a library includes searching a database.