Virtual controller for visual displays

• US 8,049,719 B2
• Filed: 10/14/2010
• Issued: 11/01/2011
• Est. Priority Date: 08/08/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sinclair, Michael J., Wilson, Andrew David
• Primary Examiner(s): Wang; Quan-Zhen
• Assistant Examiner(s): Chow; Yuk
• Application Number:
  US12/904,820
• Publication Number:
  US 20110025601A1
• Time in Patent Office: 383 Days
• Field of Search: 345/98, 345/156, 345/173, 382/103, 382/199, 382/257
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Virtual controllers for visual displays are described. In one implementation, a camera captures an image of hands against a background. The image is segmented into hand areas and background areas. Various hand and finger gestures isolate parts of the background into independent areas, which are then assigned control parameters for manipulating the visual display. Multiple control parameters can be associated with attributes of multiple independent areas formed by two hands, for advanced control including simultaneous functions of clicking, selecting, executing, horizontal movement, vertical movement, scrolling, dragging, rotational movement, zooming, maximizing, minimizing, executing file functions, and executing menu choices.

Claims

1. A method, comprising:

detecting an image of one or more skeletal body portions against a substantially stationary background via an image sensor;

segmenting the image into skeletal body portion areas and background areas;

at intervals, counting a number of independent areas of the background, wherein each part of the background that is visually independent from other parts of the background is defined as an independent area, each independent area being visually separate from the other parts of the background by at least a part of the skeletal body portion areas;

associating a control parameter for manipulating a displayed image on a user interface with a predefined region of each counted independent area; and

changing the displayed image via the control parameter in relation to each change in a shape and/or a position of the independent area associated with the control parameter.

2. The method as recited in claim 1, further comprising dissociating a control parameter from an associated independent area when the independent area becomes connected again.

3. The method as recited in claim 1, wherein the control parameter comprises emulating an actuation of a mouse button when an independent area:

forms, unforms, and reforms within a threshold time interval.

4. The method as recited in claim 1, wherein the control parameter comprises emulation and movement of a mouse pointer.

5. The method as recited in claim 1, wherein the control parameter comprises movement of a cursor.

6. The method as recited in claim 1, wherein the control parameter comprises movement of the displayed image with respect to the user interface.

7. The method as recited in claim 6, wherein the movement comprises one of:

vertical movement, horizontal movement, or rotational movement.

8. The method as recited in claim 1, wherein the control parameter comprises zooming or reducing a size of the displayed image on the user interface.

9. The method as recited in claim 1, further comprising associating the control parameter with a relation between two independent areas, each independent area formed by an image of a respective skeletal body portion.

10. The method as recited in claim 9, wherein the control parameter comprises stretching the displayed image in relation to a changing distance between the two independent areas.

11. The method as recited in claim 9, wherein the control parameter associated with the relation between two independent areas controls one of:

rotating the displayed image in relation to rotation of a line between the two independent areas;

rotating the displayed image in a clockwise or a counterclockwise direction when both independent areas move in a similar arc direction;

moving the entire displayed image when both independent areas move simultaneously in the same direction; and

zooming the displayed image when the distance between independent areas changes.

12. The method as recited in claim 1, wherein a thumb area and a forefinger area of a skeletal body portion area defines a thumb and forefinger interface (TAFFI), the thumb area and the forefinger area visually touching each other to visually disconnect a part of the background from the other parts of the background to form an independent area.

13. A system, comprising:

an image sensor to detect an image including one or more skeletal body portions against a substantially stationary background;

an image separator to segment the image into one or more skeletal body portion areas and the background;

a calculation module to count, at intervals, a number of independent areas of the background, wherein each part of the background that is visually independent from other parts of the background is defined as an independent area, each independent area being visually separated from the other parts of the background by at least a part of the skeletal body portion areas;

a linking module to associate a control parameter on a user interface with a predefined region of each counted independent area; and

wherein each change in a shape and/or a position of the independent area associated with the control parameter initiates an action via the control parameter.

14. A system as recited in claim 13, wherein when the independent area becomes connected again the linking module dissociates the respective control parameter.

15. A system as recited in claim 13, wherein the linking module associates a control parameter that emulates an actuation of a mouse button when an independent area forms, unforms, and reforms within a threshold time interval.

16. A system as recited in claim 13, wherein the linking module associates a control parameter to emulate a movement of a visual indicator selected from the group of visual indicators consisting of an image of a mouse pointer, an image of a cursor, and at least part of a displayed image on the user interface and wherein the movement comprises a movement selected from the group of movements consisting of a vertical movement, a horizontal movement, a rotational movement, a zooming-in movement, and a zooming-out movement.

17. A system as recited in claim 13, wherein the linking module associates the control parameter with a relation between two independent areas, each independent area formed by an image of a respective skeletal body portion.

18. A system as recited in claim 17, wherein the linking module associates a control parameter to stretch a displayed image on the user interface in relation to a changing distance between the two independent areas.

19. A system as recited in claim 17, wherein the linking module associates a control parameter to rotate a displayed image on the user interface in relation to a rotation of a line between the two independent areas.

20. A system for changing a displayed image on a user interface via a control parameter assigned to a movement of a skeletal body portion, the system comprising:

means for detecting an image of one or more skeletal body portions against a substantially stationary background;

means for segmenting the image into one or more skeletal body portion areas and the background;

means for counting, at fixed intervals, a number of independent areas of the background, wherein each part of the background that is visually independent from other parts of the background, and visually separated by at least a part of a skeletal body portion area from the other parts of the background, is defined as an independent area;

means for associating a control parameter on the user interface with a predefined region of each counted independent area and initiating an action via the control parameter in relation to each change in a shape and/or a position of the independent area associated with the control parameter.