Unified system for bimanual interactions on flexible representations of content

• US 10,558,341 B2
• Filed: 02/20/2017
• Issued: 02/11/2020
• Est. Priority Date: 02/20/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hinckley, Kenneth P., Buxton, William Arthur Stewart, Pahud, Michel, Xia, Haijun
• Primary Examiner(s): Bashore, William L
• Assistant Examiner(s): Shrewsbury, Nathan K
• Application Number:
  US15/437,362
• Publication Number:
  US 20180239520A1
• Time in Patent Office: 1,086 Days
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

The unified system for bimanual interactions provides a lightweight and integrated interface that allows the user to efficiently interact with and manipulate content in the user interface. The system is configured to detect a multi-finger interaction on the touchscreen and to differentiate whether the user intends to pan, zoom or frame a portion of the user interface. Generally, the framing interaction is identified by detection of the user'"'"'s thumb and forefinger on the touchscreen, which cooperate to define a focus area between vectors extending outwards from the user'"'"'s thumb and forefinger. Upon a determination that the user intends to interact with or manipulate content within the focus area, the unified system for bimanual interactions provides an indication of the objects that are located within the focus area and contextual menus for interacting with the objects.

Claims

1. A method for providing a unified system for bimanual interactions on flexible representations of content, comprising:

receiving a selection of an object that is displayed within a focus area of an application canvas, wherein the focus area is defined by a first vector extending along a first finger of a hand for a given distance, a second vector extending along a second finger of the hand for the given distance, and an arc between the first vector and the second vector;

in response to receiving the selection of the object, displaying controls in the focus area for interacting with the object, wherein;

at least some of the controls are positioned along an axis of a contextual menu,

each of the controls positioned along the axis render the object for display at a different level of formality and structure associated with the object when selected, and

the level of formality and structure of the object rendered for display is increased by selecting controls positioned in one direction from a selected control along the axis, while the level of formality and structure of the object rendered for display is decreased by selecting controls positioned in an opposite direction from the selected control along the axis;

receiving a selection of a given control indicating an interaction with the object; and

updating the object based on the interaction indicated.

2. The method of claim 1, wherein a region associated with a palm of the hand defines a fold control, and the selection of the given control is received by dragging the object to the fold control.

3. The method of claim 2, wherein the interaction indicated with the object removes the object from the application canvas and places the object onto a clipboard.

4. The method of claim 3, wherein an icon of the object is displayed in the region as a paste control, wherein a selection of the paste control inserts the object into the application canvas in the focus area.

5. The method of claim 3, wherein the clipboard is a cloud-storage location associated with a user and is configured to provide objects placed thereon for use across multiple devices.

6. The method of claim 1, wherein displaying controls in the focus area for interacting with the object further comprises:

displaying parent controls in the focus area, each of the parent controls expandable into a set of sub-controls for interacting with the object, wherein the given control selected is one of the sub-controls.

7. The method of claim 6, wherein the set of sub-controls are displayed in the focus area on an axis extending from each parent control.

8. The method of claim 7, wherein the set of sub-controls are organized into functional categories, wherein each functional category is displayed along an associated axis extending from each parent control.

9. The method of claim 1, wherein a selection of the object is received in response to defining the focus area and determining that the object is displayed within the focus area.

10. A system for providing a unified system for bimanual interactions on flexible representations of content, comprising:

a processing unit; and

a memory including instructions, which when executed by the processing unit, causes the system to;

receive a selection of an object, wherein the object is displayed within a focus area of an application canvas defined by a first vector extending along a first finger of a hand for a given distance, a second vector extending along a second finger of the hand for the given distance, and an arc between the first vector and the second vector;

in response to receiving the selection of the object, display controls in the focus area for interacting with the object, wherein;

at least some of the controls are positioned along an axis of a contextual menu,

each of the controls positioned along the axis render the object for display at a different level of formality and structure associated with the object when selected, and

the level of formality and structure of the object rendered for display is increased by selecting controls positioned in one direction from a selected control along the axis, while the level of formality and structure of the object rendered for display is decreased by selecting controls positioned in an opposite direction from the selected control along the axis;

receive a selection of a given control indicating an interaction with the object; and

update the object based on the interaction indicated.

11. The system of claim 10, wherein the first vector and the second vector intersect in the application canvas at a region associated with a palm of the hand.

12. The system of claim 11, wherein the region associated with the palm of the hand defines a fold control, and wherein the selection of the given control is received by dragging the object to the fold control.

13. The system of claim 12, wherein the interaction indicated with the object copies the object from the application canvas and places the object onto a clipboard.

14. The system of claim 13, wherein an icon of the object is displayed in the region as a paste control, wherein a selection of the paste control inserts the object into the application canvas in the focus area.

15. The system of claim 13, wherein the clipboard is a cloud-storage location associated with a user and is configured to provide objects placed thereon for use across multiple devices.

16. The system of claim 10, wherein to display controls in the focus area for interacting with the object, the system is further operative to:

display parent controls in the focus area, each of the parent controls expandable into a set of sub-controls for interacting with the object, wherein the given control selected is one of the sub-controls.

17. The system of claim 16, wherein the set of sub-controls are displayed in the focus area on an axis extending from each parent control.

18. The system of claim 17, wherein the set of sub-controls are organized into functional categories, wherein each functional category is displayed along an associated axis extending from each parent control.

19. A computer readable storage media including computer readable instructions, which when executed by a processing unit, performs steps for providing unified system for bimanual interactions on flexible representations of content, comprising:

receiving a selection of an object, wherein the object is displayed within a focus area of an application canvas defined by a first vector extending along a first finger of a hand for a given distance, a second vector extending along a second finger of the hand for the given distance, and an arc between the first vector and the second vector;

in response to receiving the selection of the object, displaying controls in the focus area for interacting with the object, wherein;

at least some of the controls are positioned along an axis of a contextual menu,

each of the controls positioned along the axis render the object for display at a different level of formality and structure associated with the object when selected, and

the level of formality and structure of the object rendered for display is increased by selecting controls positioned in one direction from a selected control along the axis, while the level of formality and structure of the object rendered for display is decreased by selecting controls positioned in an opposite direction from the selected control along the axis;

receiving a selection of a given control indicating an interaction with the object; and

updating the object based on the interaction indicated.