Virtual controller for visual displays

• US 8,115,732 B2
• Filed: 04/23/2009
• Issued: 02/14/2012
• Est. Priority Date: 08/08/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Sinclair, Michael J.
• Primary Examiner(s): Wang, Quan-Zhen
• Assistant Examiner(s): Chow, Yuk
• Application Number:
  US12/428,492
• Publication Number:
  US 20090208057A1
• Time in Patent Office: 1,027 Days
• Field of Search: 345/98, 345/156, 345/173, 382/103, 382/199, 382/257
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Virtual controllers for visual displays are described. In one implementation, a camera captures an image of hands against a background. The image is segmented into hand areas and background areas. Various hand and finger gestures isolate parts of the background into independent areas, which are then assigned control parameters for manipulating the visual display. Multiple control parameters can be associated with attributes of multiple independent areas formed by two hands, for advanced control including simultaneous functions of clicking, selecting, executing, horizontal movement, vertical movement, scrolling, dragging, rotational movement, zooming, maximizing, minimizing, executing file functions, and executing menu choices.

Claims

1. A method, comprising:

determining a state of each of a user'"'"'s hands against a substantially stationary background;

based on the determined state, detecting whether a plurality of fingers on at least one of the user'"'"'s hands are in contact with each other, the plurality of fingers forming a visually independent area of the background, the visually independent area being visually separate from other portions of the background by at least a part of at least one of the user'"'"'s hands;

associating a control parameter of a user interface to a predefined portion of the visually independent area; and

changing an image displayed on the user interface via the control parameter by changing a shape and/or position of the visually independent area.

2. The method of claim 1, wherein determining a state of each of a user'"'"'s hands includes analyzing an image captured by a camera.

3. The method of claim 2, further comprising:

detecting movement of the visually independent area closer to the camera or farther from the camera; and

changing a zoom level of the image displayed on the user interface in response to the detected movement.

4. The method of claim 1, wherein determining a state of the user'"'"'s hands comprises determining whether at least one of the user'"'"'s hands is typing or forming a visually independent area.

5. The method of claim 1, further comprising:

detecting a gesture combination including forming and unforming the visually independent area within a threshold interval of time; and

selecting an item displayed on the user interface in response to the detecting.

6. The method of claim 5, wherein detecting the gesture combination and a movement of the independent area performs at least one of:

dragging, scrolling, moving, rotating, or zooming the item displayed on the user interface.

7. A computer storage medium storing computer-executable instructions that when executed by a processor cause a computer to execute steps comprising:

detecting an image of one or more body parts against a substantially stationary background via an image sensor, the image comprising a plurality of pixels;

determining an attribute of the plurality of pixels;

segmenting the image into background pixels and other pixels according to a value of the attribute for individual pixels of the plurality of pixels;

associating a control parameter for a user interface with a region of an independent group of background pixels, the independent group of background pixels being isolated from other background pixels by at least a partial perimeter comprising other pixels; and

initiating an action on the user interface via the control parameter by modifying a composition of the independent group of background pixels.

8. The computer storage medium of claim 7, wherein the attribute includes at least one of brightness, grayscale, color component intensity, color plane value, vector pixel value, or color map index value.

9. The computer storage medium of claim 7, wherein the segmenting includes assigning a first binary value to a pixel when a value of the attribute associated with the pixel is less than a threshold and otherwise assigning a second binary value to the pixel, the first binary value being associated with background pixels and the second binary value being associated with other pixels.

10. The computer storage medium of claim 7, wherein the control parameter is associated with the independent group of background pixels when the independent group of background pixels includes at least a predetermined minimum number of pixels.

11. The computer storage medium of claim 7, wherein the region of the independent group of background pixels comprises one or more individual pixels.

12. The computer storage medium of claim 7, wherein modifying the composition of the independent group of background pixels comprises adding pixels to the independent group, removing pixels from the independent group, or a combination of adding and removing pixels from the independent group concurrently.

13. The computer storage medium of claim 7, wherein the one or more body parts comprise one or more human hands.

14. The computer storage medium of claim 7, wherein the region of the independent group of background pixels comprises a centroid of the independent group of background pixels.

15. A system comprising:

a substantially stationary background;

an image sensor configured to detect an image of one or more body parts against the substantially stationary background, the image comprising a plurality of pixels;

an image separator configured to segment the image into background pixels and other pixels according to a value of an attribute of the plurality of pixels;

a computational module configured to compute a connected component comprised of a subset of background pixels, wherein every background pixel in the subset may be reached from any other background pixel in the subset by traversing adjacent background pixels belonging to the subset;

a linking module configured to associate a control parameter for a user interface with a region of the connected component; and

a control module configured to initiate an action on the user interface in response to a change in a shape and/or a position of the connected component associated with the control parameter.

16. The system of claim 15, wherein the one or more body parts comprise a thumb and/or one or more fingers on at least one of a user'"'"'s hands.

17. The system of claim 15, wherein the linking module is configured to associate the control parameter with the connected component when the subset of background pixels includes at least a predetermined minimum number of pixels.

18. The system of claim 15, wherein the linking module is configured to associate the control parameter with a relation between more than one connected component, each connected component formed by an image of one or more respective body parts.

19. The system of claim 18, wherein the control parameter provides at least one of:

zooming in, zooming out, rotational change, or multi-directional image manipulation to an image displayed on the user interface with a change in the relation between the more than one connected component.

20. The system of claim 15, wherein the computational module is configured to change the shape and/or the position of the connected component based on a detected change to the image, by modifying a composition of the subset of background pixels.