Virtual Controller For Visual Displays

• US 20120105315A1
• Filed: 01/09/2012
• Published: 05/03/2012
• Est. Priority Date: 08/08/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Sinclair, Michael J.
• Granted Patent: US 8,552,976 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Virtual controllers for visual displays are described. In one implementation, a camera captures an image of hands against a background. The image is segmented into hand areas and background areas. Various hand and finger gestures isolate parts of the background into independent areas, which are then assigned control parameters for manipulating the visual display. Multiple control parameters can be associated with attributes of multiple independent areas formed by two hands, for advanced control including simultaneous functions of clicking, selecting, executing, horizontal movement, vertical movement, scrolling, dragging, rotational movement, zooming, maximizing, minimizing, executing file functions, and executing menu choices.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions;

detecting a user gesture against a background;

based on the detected user gesture, forming an independent area of the background that is separate from the rest of the background, the independent area being associated with the detected user gesture; and

manipulating an object displayed in a user interface in response to detecting a predetermined change of the independent area of the background.

2. The method as recited in claim 1, wherein the predetermined change of the independent area comprises a change in a position, a shape and/or an existence of the independent area.

3. The method as recited in claim 1, further comprising associating a control parameter of the user interface to a predefined portion of the independent area of the background.

4. The method as recited in claim 3, wherein manipulating the object displayed in the user interface comprises manipulating the object via the control parameter associated with the predefined portion of the independent area.

5. The method as recited in claim 1, wherein detecting the user gesture comprises capturing an image of the user gesture using an image sensor.

6. The method as recited in claim 1, further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time; and

selecting the object displayed in the user interface in response to detecting the gesture combination.

7. The method as recited in claim 1, further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time;

detecting a movement of the independent area of the background; and

in response to detecting the gesture combination and the movement of the independent area, enabling dragging, scrolling, moving, rotating or zooming of the object displayed in the user interface.

8. One or more computer storage media storing computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform acts comprising:

detecting an image of a user gesture against a background via an image sensor;

segmenting the image into an independent area associated with the user gesture and a background area corresponding to a background against which the user gesture is detected; and

manipulating an object displayed in a user interface in response to detecting a predetermined change of the independent area associated with the user gesture.

9. The one or more computer storage media as recited in claim 8, the acts further comprising associating a control parameter of the user interface to the independent area associated with the user gesture.

10. The one or more computer storage media as recited in claim 9, wherein segmenting the image comprises:

determining an attribute of a plurality of pixels of the image; and

assigning a first value to a pixel of the plurality of pixels if a value of the attribute associated with the pixel is less than a threshold and assigning a second value to the pixel otherwise.

11. The one or more computer storage media as recited in claim 10, wherein the attribute comprises brightness, grayscale, color component intensity, color plane value, vector pixel value or color map index value.

12. The one or more computer storage media as recited in claim 8, wherein detecting the predetermined change of the independent area comprises detecting a change in the number of pixels associated with the independent area.

13. The one or more computer storage media as recited in claim 8, wherein the independent area comprises at least a predetermined minimum number of pixels in the image.

14. A system comprising:

one or more processors;

memory, communicatively coupled to the one or more processors, storing instructions that, when executed by the one or more processors, configure the one or more processors to perform acts comprising;

detecting a user gesture against a background;

based on the detected user gesture, forming an independent area of the background that is separate from the rest of the background, the independent area being associated with the detected user gesture; and

manipulating an object displayed in a user interface in response to detecting a predetermined change of the independent area of the background.

15. The system as recited in claim 14, wherein the predetermined change of the independent area comprises a change in a position, a shape and/or an existence of the independent area.

16. The system as recited in claim 14, the acts further comprising associating a control parameter of the user interface to a predefined portion of the independent area of the background.

17. The system as recited in claim 16, wherein manipulating the object displayed in the user interface comprises manipulating the object via the control parameter associated with the predefined portion of the independent area.

18. The system as recited in claim 14, wherein detecting the user gesture comprises capturing an image of the user gesture using an image sensor.

19. The system as recited in claim 14, the acts further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time; and

selecting the object displayed in the user interface in response to detecting the gesture combination.

20. The system as recited in claim 14, the acts further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time;

detecting a movement of the independent area of the background; and

in response to detecting the gesture combination and the movement of the independent area, enabling dragging, scrolling, moving, rotating or zooming of the object displayed in the user interface.