POSITION AWARE GESTURES WITH VISUAL FEEDBACK AS INPUT METHOD

• US 20120110516A1
• Filed: 10/28/2010
• Published: 05/03/2012
• Est. Priority Date: 10/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tumanov, Ilya
• Granted Patent: US 9,195,345 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04W 4/21 for social networking applications

Abstract

A gesture based user interface is provided for a user to interact with a device in order to operate and control the device through detection of gestures and movements of the user. Visual feedback of the user gestures is provided to the user to aid in the user'"'"'s operational and control decisions of a device. An image capturing device such as a video camera may be employed to capture a user'"'"'s image, and an integrated application on a computing device may process continuous images from the capturing device to recognize and track user gestures. The gestures may correlate to an object and/or location on the display and the user'"'"'s image may be projected on the display to provide visual feedback of the user'"'"'s interaction.

Claims

1. A method for providing input to a computer system comprising:

capturing one or more images of a user creating an input gesture;

processing the images to track the input gesture;

determining a relative location of the user gesture on a user interface;

providing visual feedback of the user gesture over the user interface; and

enabling the user to control an application through the gesture.

2. The method of claim 1, wherein the input gesture includes at least one from a set of:

a motion of a body part, a positioning of a body part, a motion of an entire body, a positioning of an entire body.

3. The method of claim 1, wherein the visual feedback includes displaying of at least one of a scaled version and a transformed version of the captured image of the user near an element of the user interface focused on by the user.

4. The method of claim 3, wherein the captured image of the user is displayed in one of a transparent and semi-transparent manner such that interference with the user interface is reduced.

5. The method of claim 1, further comprising:

capturing images of a plurality of gestures performed by a plurality of body parts;

providing visual feedback of the plurality of gestures over the user interface such that the user is enabled to control a plurality of aspects of the application through the plurality of gestures.

6. The method of claim 1, further comprising:

capturing images of a plurality of gestures performed by a plurality of users;

providing visual feedback of the plurality of gestures over the user interface such that the users are enabled to control a plurality of aspects of the application through the plurality of gestures.

7. The method of claim 6, wherein the users are enabled to distinguish their respective gestures based on displayed images on the user interface.

8. The method of claim 1, wherein the gesture further includes one of a sign language gesture and a gesture representing a standardized textual character.

9. The method of claim 1, wherein the application is controlled by one of:

an input module integrated to the application, an input module integrated to an operating system of the computer system, and an input control application executed by the computer system.

10. An electronic device capable of accepting position aware gesture based input, the electronic device comprising:

a functional component;

an image capture component;

a display; and

a control component configured to;

capture images of a user performing a gesture;

compute a difference between a current frame and a previous frame;

determine a location of changed areas on the captured images based on the difference;

determine at least one of;

a size, a shape, and a position of objects in the change areas;

determine trajectories of the objects;

determine the gesture based on a comparison of the trajectories of the objects with preset patterns and location of the gesture as captured by the image capture component; and

employ the determined gesture to control an aspect of the functional component.

11. The electronic device of claim 10, wherein the control component is further configured to:

generate a binary image from the captured images;

convert a semi-transparent image from the binary image;

scale the semi-transparent image;

overlay the semi-transparent image onto a user interface depicted on the display.

12. The electronic device of claim 11, wherein the control component is further configured to:

transform the semi-transparent image to match the displayed user interface; and

match coordinates of the captured image with coordinates of the user interface depicted on the display such that the visual feedback indicates to the user which element of the user interface is being controlled by the gesture.

13. The electronic device of claim 10, wherein the object trajectories are computed in sets of coordinates changing in time.

14. The electronic device of claim 11, wherein the control component is further configured to:

match objects in a current frame to objects detected on previous frames based on size and location proximity.

15. The electronic device of claim 10, wherein the control component is configured to determine the location of changed areas on the captured images by dividing the images into tiles of predefined sizes and counting changed pixels within each tile.

16. The electronic device of claim 10, wherein the device includes one of a computing device, a television set, an entertainment center, an interactive message board, an advertisement stand, and an automated teller machine.

17. A computer-readable storage medium with instructions stored thereon for tracking gestures and providing visual feedback, the instructions comprising:

receiving a continuous stream of images from an image capturing device;

processing the images to detect and track gestures;

correlating a relative location of each gesture to a corresponding location on a user interface display;

generating representative image of the user and projecting the image on the user interface display to provide visual feedback to the user; and

interpreting the gestures as input signals for operating aspects of an application executed on a computing device.

18. The computer-readable medium of claim 17, wherein gestures are detected by computing a difference between frames in the continuous stream of images and analyzing number of changed pixels in predefined sets of tiles in the frames.

19. The computer-readable medium of claim 17, wherein the representative image is one of a representation of the user and a representation of a body part performing the gesture, and wherein the location on the user interface is determined by matching a capturing device'"'"'s location coordinate system to the user interface display coordinate system.

20. The computer-readable medium of claim 17, wherein the gesture includes at least one from a set of:

a motion of a body part, a positioning of a body part, a motion of an entire body, a positioning of an entire body representing at least one from a set of;

a selection of a user interface option, a sign language gesture, and a standardized textual character.