Multi-modal interaction on multi-touch display

• US 8,487,888 B2
• Filed: 12/04/2009
• Issued: 07/16/2013
• Est. Priority Date: 12/04/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wigdor, Daniel J., Hoover, Paul Armistead, Hofmeester, Kay
• Primary Examiner(s): LAMB, CHRISTOPHER RAY
• Application Number:
  US12/631,609
• Publication Number:
  US 20110134047A1
• Time in Patent Office: 1,320 Days
• Field of Search: None
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Embodiments are disclosed herein that relate to multi-modal interaction on a computing device comprising a multi-touch display. One disclosed embodiment comprises a method of multi-modal interaction including recognizing a hand posture of a user'"'"'s first hand directed at the display and displaying a modal region based on the hand posture, wherein the modal region defines an area on the display. The method further includes receiving an input selecting a mode to be applied to the modal region, wherein the mode indicates functionalities to be associated with the modal region and defines a mapping of touch gestures to actions associated with the mode. The method further includes, while the modal region remains displayed, recognizing a touch gesture from a user'"'"'s second hand directed at the display within the modal region and performing an action on the display based upon a mapping of the touch gesture.

Claims

1. On a computing system comprising a multi-touch display, a method of multi-modal interaction, the method comprising:

displaying on the display a first content item and a second content item;

recognizing a hand posture of a user'"'"'s first hand directed at the display;

displaying on the display a modal region based on the hand posture, the modal region defining an area on the display at a first location on the display based upon a location of the hand posture;

applying a mode to the modal region, the mode indicating a family of functionalities to be associated with the modal region and defining a mapping of one or more touch gestures to one or more corresponding actions associated with the mode;

while the modal region remains displayed, recognizing a first touch gesture interacting with the first content item within the modal region at the first location on the display;

performing a first action on the display based upon a mapping of the first touch gesture to the first action;

,

detecting a change in the location of the hand posture;

in response, displaying a movement of the modal region to a second location on the display such that the modal region moves over the second content item;

recognizing a second touch gesture interacting with the second content item within the modal region at the second location on the display; and

performing a second action on the display based upon a mapping of the second touch gesture to the second action.

2. The method of claim 1, further comprising detecting a hand posture of a second user'"'"'s first hand directed at the display, and in response, displaying on the display a second modal region.

3. The method of claim 2, wherein the hand posture of the second user'"'"'s first hand and the hand posture of the user'"'"'s first hand are temporally overlapping.

4. The method of claim 1, further comprising detecting lifting of the user'"'"'s first hand, and in response, continuing to display the modal region after lifting is detected.

5. The method of claim 1, further comprising detecting lifting of the user'"'"'s first hand, and in response, ceasing display of the modal region.

6. The method of claim 1, further comprising receiving a locking input, detecting lifting of the user'"'"'s first hand after receiving the locking input, and continuing display of the modal region after the user'"'"'s first hand is lifted.

7. The method of claim 1, wherein applying the mode comprises applying a default mode to the modal region.

8. The method of claim 1, wherein applying the mode includes receiving an input selecting the mode via a touch input directed at a virtual control displayed on the display.

9. In a display device comprising a multi-touch display, a computing device comprising:

a processor; and

memory comprising instructions stored thereon that are executable by the processor to;

display on the display a first content item and a second content item;

recognize a hand posture of a hand touching the display;

display on the display a modal region based on the hand posture, the modal region defining an area on the display and being displayed at a first location on the display based on a location of the hand posture;

receive an input selecting a mode to be applied to the modal region, the mode indicating a family of functionalities to be associated with the modal region and defining a mapping of one or more touch gestures to one or more corresponding actions associated with the mode;

recognize a first touch gesture interacting with the first content item within the modal region at the first location on the display;

perform a first action on the display based upon a mapping of the first touch gesture to the first action;

detect a change in the location of the hand posture and in response, adjust the location of the modal region to a second location on the display such that the modal region moves over the second content item;

while the modal region remains displayed, recognize a second touch gesture from a second hand directed at the second content item within the modal region at the second location on the display;

perform a second action on the display based upon the mapping of the second touch gesture to the second action; and

recognize a removal of the hand posture from the display and in response, remove display of the modal region from the display.

10. The computing device of claim 9, wherein the instructions are further executable to determine that the modal region is to remain displayed if the hand posture of the hand is still touching the display device.

11. The computing device of claim 9, wherein the instructions are further executable to recognize a plurality of hand postures, and to display on the display a second modal region associated with each hand posture recognized.

12. The computing device of claim 11, wherein the modal region is a first modal region, and wherein the instructions are further executable to detect a potential overlap of the first modal region and the second modal region displayed on the display, to determine whether displaying the first modal region and the second modal region as overlapping is allowed or unallowed based upon a comparison of the mode applied to the first modal region and a mode applied to the second modal region, and to display the first modal region and the second modal region as overlapping or not overlapping depending upon the comparison.

13. The computing device of claim 9, wherein the instructions are executable to perform the first action on the display by changing a property of content displayed on the display within the modal region.

14. The computing device of claim 9, wherein the instructions are executable to perform the first action on the display by changing a property of content displayed on the display outside of the modal region.

15. In a computing system comprising a multi-touch display, a method of interacting with a plurality of users, the method comprising:

displaying a first content item and a second content item on the display;

recognizing a hand posture of a first hand of a first user directed at the display;

in response to recognizing the hand posture of the first hand of the first user, displaying on the display a first modal region based on the hand posture of the first hand of the first user, the first modal region defining an area on the display and being displayed at a first location on the display based upon a location of the hand posture;

receiving an input selecting a mode to be applied to the first modal region, the mode indicating a family of functionalities to be associated with the first modal region and defining a mapping of one or more touch gestures to one or more corresponding actions associated with the mode;

while the first modal region remains displayed, recognizing a touch gesture from a second hand of the first user directed at the first content item within the first modal region, and in response, performing a first action on the display based upon a mapping of the touch gesture from the second hand of the first user to the first action;

detecting a change in the location of the hand posture, and in response adjusting the location of the first modal region to a second location on the display such that the first modal region moves over the second content item;

while the first modal region remains displayed, recognizing a second touch gesture from a second hand directed at the second content item within the first modal region at the second location on the display;

perform a second action on the display based upon the mapping of the second touch gesture to the second action;

while the first modal region remains displayed, recognizing a hand posture of a first hand of a second user directed at the display;

in response to recognizing the hand posture of the first hand of the second user, displaying on the display a second modal region;

receiving an input selecting a mode to be applied to the second modal region; and

while the second modal region remains displayed, recognizing a third touch gesture directed at the display within the second modal region, and in response, performing a third action on the display based upon a mapping of the third touch gesture to the third action.

16. The method of claim 15, wherein the mode applied to the first modal region and the mode applied to the third modal region are a same mode.

17. The method of claim 15, wherein the mode applied to the first modal region and the mode applied to the third modal region are different modes.

18. The method of claim 15, further comprising displaying the second modal region at a location on the display based on a location of the hand posture of the first hand of the second user.