TECHNIQUES FOR USING HUMAN GESTURES TO CONTROL GESTURE UNAWARE PROGRAMS

• US 20110055846A1
• Filed: 08/31/2009
• Published: 03/03/2011
• Est. Priority Date: 08/31/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tsunoda, Kudo, Kipman, Alex A., Geisner, Kevin A., Perez, Kathryn S.
• Granted Patent: US 9,141,193 B2
• Time in Patent Office: Days
• US Class Current: 719/313
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A capture device can detect gestures made by a user. The gestures can be used to control a gesture unaware program.

Claims

1. A system, comprising:

a computing environment including a processor coupled to a computer readable storage medium, the computer readable storage medium including executable instructions for a program, wherein a user interface for the program is configured to only receive non-gesture user interface commands;

a capture device coupled to the computing environment, wherein the capture device is configured to capture images and identify gestures in the images, wherein the capture device is configured to capture audio and identify audio commands in the captured audio; and

wherein the non-gesture user interface commands used to control the user interface of the program are generated from gestures identified in the images and audio commands identified in the captured audio.

2. The system of claim 1, wherein the capture device is further configured to generate scancodes for keys of keyboards based on the gestures and send the scancodes to the computing environment.

3. The system of claim 1, wherein the capture device is further configured to generate packets that include mouse button state and mouse position information based on the gestures and send the packets to the computing environment.

4. The system of claim 1, wherein the capture device is configured to generate state information for a videogame controller and send the state information to the computing environment.

5. The system of claim 1, wherein the computing environment is configured to receive gesture information, generate keyboard messages, and send the keyboard messages to the program.

6. The system of claim 1, wherein the computing environment is configured to receive gesture information, generate mouse messages, and send the mouse messages to the program.

7. The system of claim 1, wherein the computing environment is configured to receive gesture information, generate state information for a videogame controller, and send the state information for the videogame controller to the program.

8. A computer readable storage medium including executable instructions, the computer readable storage medium comprising:

instructions for identifying a gesture from an image captured by a camera;

instructions for generating a command for a tactile based user input device based on the identified gesture; and

instructions for sending the command to a gesture unaware program configured to process the command.

9. The computer readable storage medium of claim 8, wherein the instructions for identifying the gesture in the image further comprise:

instructions for comparing image data generated from the image to a plurality of filters, each filter associated with a different gesture;

instructions for generating a confidence score for each comparison;

instructions for selecting the gesture associated with the filter having the highest confidence score.

10. The computer readable storage medium of claim 8, wherein the instructions for generating the command further comprise:

instructions for identifying a keyboard command associated with the gesture;

instructions for generating a keyboard message including the keyboard command; and

instructions for sending the keyboard message to the program.

11. The computer readable storage medium of claim 8, wherein the instructions for generating the command further comprise:

instructions for identifying a mouse status associated with the gesture; and

instructions for sending information indicative of the mouse status to the program.

12. The computer readable storage medium of claim 8, wherein the instructions for generating the command further comprise:

instructions for identifying a videogame controller status associated with the gesture; and

instructions for sending information indicative of the videogame controller status to the program.

13. The computer readable storage medium of claim 8, further comprising:

instructions for determining that the program registered for specific tactile-based user input; and

instructions for loading a gesture library that maps gestures to the specific tactile-based user input.

14. The computer readable storage medium of claim 8, further comprising:

instructions for executing a plurality of programs, each program associated with a window and different gestures;

instructions for determining that the program is the currently focused window; and

instructions for mapping gestures associated with the program to tactile based user input device commands.

15. A device method, comprising:

receiving image data from a camera;

identifying a gesture from the image data;

generating a command for a tactile based user input device based on the identified gesture; and

sending the command to a computing environment coupled to the device.

16. The method of claim 15, wherein generating the command further comprises:

generating a packet of information, the packet of information formatted to emulate a packet generated by a mouse.

17. The method of claim 15, wherein generating the command further comprises:

generating a scancode indicative of a keyboard action, the scancode formatted to emulate a scancode generated by a keyboard.

18. The method of claim 15, wherein generating the command further comprises:

generating a packet of information, the packet of information formatted to emulate a packet generated by a videogame controller.

19. The method of claim 15, wherein identifying the gesture further comprises:

comparing the image to a plurality of filters, each filter associated with a different gesture;

generating a confidence score for each comparison; and

selecting the gesture associated with the filter having the highest confidence score.

20. The method of claim 15, further comprising:

determining that a program running on the computing environment is gesture unaware; and

loading a gesture library that maps gestures to tactile based user input device commands.