Techniques for using human gestures to control gesture unaware programs

• US 9,141,193 B2
• Filed: 08/31/2009
• Issued: 09/22/2015
• Est. Priority Date: 08/31/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Perez, Kathryn S., Geisner, Kevin A., Kipman, Alex A., Tsunoda, Kudo
• Primary Examiner(s): Salomon, Phenuel
• Assistant Examiner(s): Nguyen, Tuan S
• Application Number:
  US12/551,370
• Publication Number:
  US 20110055846A1
• Time in Patent Office: 2,213 Days
• Field of Search: 704/243, 715/856, 715/863, 245/156, 382/103
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A capture device can detect gestures made by a user. The gestures can be used to control a gesture unaware program.

Claims

1. A computer readable hardware device, including executable instructions, the computer readable storage device comprising:

instructions for receiving first image data from a camera, the first image data representing a user motion or pose;

instructions for, in response to determining that a first application is configured to process gesture data, sending the first application an output indicative of whether the user motion or pose is indicative of performing a gesture;

instructions for receiving second image data, the second image data representing a user motion or pose;

instructions for, before a second application has received an indication of the second image data, and in response to determining that the second application is not configured to process gesture data, and in response to changing input focus to the second application that is not configured to process gesture data, replacing a first gesture library with a second gesture library that outputs tactile based user commands and using the second gesture library to generate a command for a tactile based user input device based on the second image data, the second gesture library being configured to send tactile based user commands to a plurality of applications that includes the second application; and

instructions for sending the command generated using the second gesture library to the second application.

2. The computer readable hardware device of claim 1, wherein the instructions for, in response to determining that the first application is configured to process gesture data, sending the first application the output indicative of whether the user motion or pose further comprise:

instructions for sending the first application an indication of a confidence level that the user motion or pose corresponds to performing a gesture.

3. The computer readable hardware device of claim 1, wherein the instructions for generating the command further comprise:

instructions for identifying a keyboard command associated with the motion or pose;

instructions for generating a keyboard message including the keyboard command; and

instructions for sending the keyboard message to the second application.

4. The computer readable hardware device of claim 1, wherein the instructions for generating the command further comprise:

instructions for identifying a mouse status associated with the motion or pose; and

instructions for sending information indicative of the mouse status to the second application.

5. The computer readable hardware device of claim 1, wherein the instructions for generating the command further comprise:

instructions for identifying a videogame controller status associated with the motion or pose; and

instructions for sending information indicative of the videogame controller status to the second application.

6. The computer readable hardware device of claim 1, further comprising:

instructions for determining that the second application registered for specific tactile-based user input; and

instructions for loading a gesture library that maps gestures to the specific tactile-based user input.

7. The computer readable hardware device of claim 1, wherein the instructions for, in response to determining that a first application is configured to process gesture data, sending the first application an output indicative of whether the user motion or pose;

further comprise;

instructions for sending the first application the output in response to determining that the first application has the currently focused window.

8. A method, comprising:

receiving first image data from a camera, the first image data representing a user motion or pose;

in response to determining that a first application is configured to process gesture data, sending the first application an output indicative of whether the user motion or pose is indicative of performing a gesture;

receiving second image data, the second image data representing a user motion or pose;

before a second application has received an indication of the second image data, in response to determining that the second application is not configured to process gesture data, and in response to changing input focus to the second application that is not configured to process gesture data, replacing a first gesture library with a second gesture library that outputs tactile based user commands and using the second gesture library to generate a command for a tactile based user input device based on the second image data, the second gesture library being configured to send tactile based user commands to a plurality of applications that includes the second application; and

sending the command to the second application.

9. The method of claim 8, wherein generating the command further comprises:

generating a packet of information, the packet of information formatted to emulate a packet generated by a mouse.

10. The method of claim 8, wherein generating the command further comprises:

generating a scancode indicative of a keyboard action, the scancode formatted to emulate a scancode generated by a keyboard.

11. The method of claim 8, wherein generating the command further comprises:

generating a packet of information, the packet of information formatted to emulate a packet generated by a videogame controller.

12. The method of claim 8, wherein in response to determining that the first application is configured to process gesture data, sending the first application the output indicative of whether the user motion or pose further comprises:

sending the first application an indication of a confidence level that the user motion or pose corresponds to performing a gesture.

13. The method of claim 8, further comprising:

in response to determining that the second application is not configured to process gesture data, loading a gesture library that maps gestures to tactile based user input device commands; and

wherein generating the command for the tactile based user input device based on the second image data comprises;

determining the command based on using the second image data as input to the gesture library.

14. A system, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system at least to;

receive first image data from a camera, the first image data representing a user motion or pose;

in response to determining that a first application is configured to process gesture data, using a first gesture library that outputs gesture data to send the first application an output indicative of whether the user motion or pose is indicative of performing a gesture;

receive second image data, the second image data representing a user motion or pose;

before a second application has received an indication of the second image data, and in response to changing input focus to the second application that is not configured to process gesture data, replacing the first gesture library with a second gesture library that outputs tactile based user commands and using the second gesture library to generate a command for a tactile based user input device based on the second image data, the second gesture library being configured to send tactile based user commands to a plurality of applications that includes the second application; and

send the command to the second application.

15. The system of claim 14, wherein the processor-executable instructions that, when executed on the processor, cause the system at least to generate a command for a tactile based user input device based on the second image data further cause the system at least to:

generate scancodes for keys of keyboards.

16. The system of claim 14, wherein the processor-executable instructions that, when executed on the processor, cause the system at least to generate a command for a tactile based user input device based on the second image data further cause the system at least to:

generate packets that include mouse button state and mouse position information.

17. The system of claim 14, wherein the processor-executable instructions that, when executed on the processor, cause the system at least to generate a command for a tactile based user input device based on the second image data further cause the system at least to:

generate state information for a videogame controller.

18. The system of claim 14, wherein the processor-executable instructions that, when executed on the processor, cause the system at least to generate a command for a tactile based user input device based on the second image data further cause the system at least to:

generate keyboard messages.

19. The system of claim 14, wherein the processor-executable instructions that, when executed on the processor, cause the system at least to generate a command for a tactile based user input device based on the second image data further cause the system at least to:

generate mouse messages.