Virtual controller for visual displays

• US 8,552,976 B2
• Filed: 01/09/2012
• Issued: 10/08/2013
• Est. Priority Date: 08/08/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Sinclair, Michael J.
• Primary Examiner(s): Wang, Quan-Zhen
• Assistant Examiner(s): Chow, Yuk
• Application Number:
  US13/346,472
• Publication Number:
  US 20120105315A1
• Time in Patent Office: 638 Days
• Field of Search: 345/98, 345/156, 345/173, 382/103, 382/199, 382/257, 382/288, 348/211.1, 715/860
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Virtual controllers for visual displays are described. In one implementation, a camera captures an image of hands against a background. The image is segmented into hand areas and background areas. Various hand and finger gestures isolate parts of the background into independent areas, which are then assigned control parameters for manipulating the visual display. Multiple control parameters can be associated with attributes of multiple independent areas formed by two hands, for advanced control including simultaneous functions of clicking, selecting, executing, horizontal movement, vertical movement, scrolling, dragging, rotational movement, zooming, maximizing, minimizing, executing file functions, and executing menu choices.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions;

detecting a user gesture against a background;

based on the detected user gesture, forming an independent area of the background that is separate from the rest of the background, the independent area being associated with the detected user gesture; and

in response to detecting a predetermined change of the independent area that is maintained to be separate from the rest of the background during a movement of the detected user gesture, performing a function with an object displayed in a user interface, wherein the predetermined change comprises at least one of a change in a position of the independent area or a change in a shape of the independent area.

2. The method as recited in claim 1, wherein the predetermined change of the independent area further comprises a change in an existence of the independent area.

3. The method as recited in claim 1, further comprising associating a control parameter of the user interface to a predefined portion of the independent area of the background.

4. The method as recited in claim 3, wherein the function that is performed with the object comprises manipulating the object via the control parameter associated with the predefined portion of the independent area.

5. The method as recited in claim 1, wherein detecting the user gesture comprises capturing an image of the user gesture using an image sensor.

6. The method as recited in claim 1, further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time; and

selecting the object displayed in the user interface in response to detecting the gesture combination.

7. The method as recited in claim 1, further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time;

detecting a movement of the independent area of the background; and

in response to detecting the gesture combination and the movement of the independent area, enabling dragging, scrolling, moving, rotating or zooming of the object displayed in the user interface.

8. One or more computer storage media storing computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform acts comprising:

detecting an image of a user gesture against a background via an image sensor;

segmenting the image into an independent area associated with the user gesture and a background area corresponding to the background against which the user gesture is detected; and

in response to detecting a change in a number of pixels associated with the independent area of the background area during a movement of the user gesture and while the independent area is maintained to be separate from the background area, performing a function with an object displayed in a user interface.

9. The one or more computer storage media as recited in claim 8, the acts further comprising associating a control parameter of the user interface to the independent area associated with the user gesture.

10. The one or more computer storage media as recited in claim 9, wherein segmenting the image comprises:

determining an attribute of a plurality of pixels of the image; and

assigning a first value to a pixel of the plurality of pixels if a value of the attribute associated with the pixel is less than a threshold and assigning a second value to the pixel otherwise.

11. The one or more computer storage media as recited in claim 10, wherein the attribute comprises brightness, grayscale, color component intensity, color plane value, vector pixel value or color map index value.

12. The one or more computer storage media as recited in claim 8, wherein the independent area comprises at least a predetermined minimum number of pixels in the image.

13. The one or more computer storage media as recited in claim 8, wherein the independent area comprises a first independent area and the user gesture comprises a first user gesture, and wherein the acts further comprise:

detecting an image of a second user gesture against the background via the image sensor;

segmenting a second independent area associated with the second user gesture from the image of the second user gesture; and

manipulating the object displayed in the user interface in response to detecting a predetermined relative change of the second independent area associated with the second user gesture with respect to the first independent area associated with the first user gesture.

14. A system comprising:

one or more processors;

memory, communicatively coupled to the one or more processors, storing instructions that, when executed by the one or more processors, configure the one or more processors to perform acts comprising;

detecting a user gesture against a background;

based on the detected user gesture, forming an independent area of the background that is separate from the rest of the background, the independent area being associated with the detected user gesture; and

in response to detecting a predetermined change of the independent area of the background during a movement of the detected user gesture and while the independent area is maintained to be separate from the rest of the background, performing a function with an object displayed in a user interface, wherein the predetermined change of the independent area comprises at least one of a change in a position of the independent area or a change in a shape of the independent area.

15. The system as recited in claim 14, wherein the predetermined change of the independent area further comprises a change in an existence of the independent area.

16. The system as recited in claim 14, the acts further comprising associating a control parameter of the user interface to a predefined portion of the independent area of the background.

17. The system as recited in claim 16, wherein the function that is performed with the object comprises manipulating the object via the control parameter associated with the predefined portion of the independent area.

18. The system as recited in claim 14, wherein detecting the user gesture comprises capturing an image of the user gesture using an image sensor.

19. The system as recited in claim 14, the acts further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time; and

selecting the object displayed in the user interface in response to detecting the gesture combination.

20. The system as recited in claim 14, the acts further comprising:

detecting a gesture combination including forming and unforming the independent area within a predetermined interval of time;

detecting a movement of the independent area of the background; and

in response to detecting the gesture combination and the movement of the independent area, enabling dragging, scrolling, moving, rotating or zooming of the object displayed in the user interface.