Position aware gestures with visual feedback as input method

• US 9,195,345 B2
• Filed: 10/28/2010
• Issued: 11/24/2015
• Est. Priority Date: 10/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Tumanov, Ilya
• Primary Examiner(s): ULRICH, NICHOLAS S
• Application Number:
  US12/914,994
• Publication Number:
  US 20120110516A1
• Time in Patent Office: 1,853 Days
• Field of Search: 715/863, 715/754, 715/856, 382/103, 382/107
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04W 4/21 for social networking applications

Abstract

A gesture based user interface is provided for a user to interact with a device in order to operate and control the device through detection of gestures and movements of the user. Visual feedback of the user gestures is provided to the user to aid in the user'"'"'s operational and control decisions of a device. An image capturing device such as a video camera may be employed to capture a user'"'"'s image, and an integrated application on a computing device may process continuous images from the capturing device to recognize and track user gestures. The gestures may correlate to an object and/or location on the display and the user'"'"'s image may be projected on the display to provide visual feedback of the user'"'"'s interaction.

Claims

1. A method for providing input to a computer system comprising:

capturing one or more images of a user creating an input gesture;

processing the captured one or more images to track the input gesture;

determining a relative location of the input gesture on a user interface by matching coordinates of the one or more images with coordinates of the user interface;

generating a binary image of the one or more images, wherein the binary image calculates a difference between frames resulting in a binary output of changed and unchanged pixels such that changed pixels indicate a user motion;

providing real-time visual feedback of the input gesture over the user interface by overlaying the binary image on the user interface such that only the changed pixels indicating the motion of the user are visible and motionless portions of the binary image are invisible;

suppressing the real-time visual feedback until a predefined gesture is performed to avoid interference;

adjusting the binary image to match a display such that the binary image is scaled larger than a screen to allow for the input gesture on edges; and

enabling the user to control an application through the input gesture.

2. The method of claim 1, wherein the input gesture includes at least one from a set of:

a motion of a body part, a positioning of a body part, a motion of an entire body, a positioning of an entire body, and eye tracking.

3. The method of claim 1, wherein the real-time visual feedback includes displaying of at least one of a scaled version and a transformed version of the captured one or more images of the user near an element of the user interface focused on by the user.

4. The method of claim 3, wherein the captured one or more images of the user is displayed in one of a transparent and semitransparent manner such that the interference with the user interface is reduced.

5. The method of claim 1, further comprising:

capturing the one or more images of the input gesture performed by a plurality of body parts; and

providing visual feedback of the input gesture over the user interface such that the user is enabled to control a plurality of aspects of the application through the input gesture.

6. The method of claim 1, further comprising:

capturing the one or more images of the input gesture performed by a plurality of users; and

providing visual feedback of the input gesture over the user interface such that the plurality of users are enabled to control a plurality of aspects of the application through the input gesture.

7. The method of claim 6, wherein the plurality of users are enabled to distinguish their respective gestures based on the one or more images on the user interface.

8. The method of claim 1, wherein the input gesture further includes one of a sign language gesture and a gesture representing a standardized textual character.

9. The method of claim 1, wherein the application is controlled by one of:

an input module integrated to the application, an input module integrated to an operating system of the computer system, and an input control application executed by the computer system.

19. The method of claim 7, wherein the representative image is one of a representation of the users and a representation of a body part performing the input gestures, and wherein the location on the user interface is determined by matching a capturing device'"'"'s location coordinate system to the user interface display coordinate system.

10. An electronic device capable of accepting position aware gesture based input, the electronic device comprising:

a functional component;

an image capture component;

a display; and

a control component executing a gesture module integrated with an application operated on the electronic device, the control component configured to;

capture images of users creating input gestures;

process the captured images to track the input gestures;

store the captured images in a memory;

divide image frames into tiles of a predefined size;

compute a difference between a current frame and a previous frame by counting changed pixels within a selected tile, wherein the changed pixels correspond to a motion of each of the users;

determine a location of changed areas on the captured images based on the difference and by combining changes in neighboring;

tiles together to produce a list of changed objects, wherein the changed objects include at least one of;

a size, a shape, and a position of objects in the change areas, representing the motion of each of the users;

determine trajectories of the objects as a set of movement vectors calculated from a set of coordinates changing in time, wherein the set of coordinates indicate the size, the shape, and the position of the changed objects within as coordinate system of the image capture component;

determine the input Restores based on a comparison of the trajectories of the objects with preset patterns and the location of the input gestures as captured by the image capture component relative to preset movement vectors within the coordinate system of the image capture component;

determine a relative location of each of the input gestures on the display by matching coordinates of the image capturing component with coordinates of the display;

generate a binary image of the captured images, wherein the binary image calculates a difference between frames resulting in a binary output of changed and unchanged pixels such that changed pixels indicate the motion of each of the users;

provide real-time visual feedback of the input gestures over the display by overlaying the binary image on a user interface of the display such that only the changed pixels indicating the motion of each of the users are visible and motionless portions of the binary image are invisible;

suppress the real-time visual feedback until predefined gesture is performed to avoid interference;

adjust the binary image to match the user interface such that the binary image is scaled larger than a screen to allow for the input gestures on edges; and

employ the input gestures to enable the users to control an aspect of the functional component.

11. The electronic device of claim 10, wherein the control component is further configured to:

convert a semi-transparent image from the binary image;

scale the semi-transparent image; and

overlay the semi-transparent image onto the user interface depicted on the display to provide the real-time visual feedback of the input gestures over the user interface.

12. The electronic device of claim 11, wherein the control component is further configured to:

transform the binary image into the semi-transparent image to match the user interface; and

match coordinates of the captured images with coordinates of the user interface depicted on the display such that the real-time visual feedback indicates to the users which element of the user interface is being, controlled by the input gestures.

13. The electronic device of claim 10, wherein the object trajectories are computed in sets of coordinates changing in time.

14. The electronic device of claim 11, wherein the control component is further configured to:

match objects in the current frame to objects detected on previous frames based on a site and a proximity to the location.

15. The electronic device of claim 10, wherein the control component is configured to determine the location of changed areas on the captured images by dividing the captured images into tiles of predefined sizes and counting the changed pixels within each tile.

16. The electronic device of claim 10, wherein the device includes one of a computing device, a television set, an entertainment center, an interactive message hoard, an advertisement stand, and an automated teller machine.

17. A method to be executed on a computing device, the method comprising:

receiving a continuous stream of images from an image capturing device, wherein the images include users creating input gestures;

processing the images to track the input gestures;

correlating a relative location of the input gestures to a corresponding location on a user interface display by matching coordinates of the image capturing device with coordinates of the user interface display;

generating, a binary image of the images, wherein the binary image calculates a difference between frames resulting in a binary output of changed and unchanged pixels such that changed pixels indicate a motion of each of the users;

providing real-time visual feedback of the input gestures over the user interface by overlaying the binary image on the user interface such that only the changed pixels indicating the motion of the users are visible and motionless portions of the binary image are invisible;

interpreting the input gestures as input signals to enable the users to operate aspects of an application executed on the computing device, wherein the input signals include one or more actions to position and move elements on the display, scroll through pages, and select objects on the display;

suppressing the real-time visual feedback until a predefined gesture is performed and again after the predefined gesture is completed to avoid interference; and

adjusting the binary image to match a display such that the binary image is scaled larger than a screen to allow for the input gestures on edges.

18. The method of claim 17, wherein the input gestures are detected by computing, a difference between frames in the continuous stream of images and analyzing a number of changed pixels in predefined sets of tiles in the frames.

20. The method of claim 17, wherein the input gestures includes at least one from a set of a motion of a body part, a positioning of a body part, a motion of an entire body, a positioning of an entire body representing at least one from a set of:

a selection of a user interface option, a sign language gesture, and a standardized textual character.