Thumb and pen interaction on a mobile device

• US 10,635,291 B2
• Filed: 02/20/2017
• Issued: 04/28/2020
• Est. Priority Date: 02/20/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hinckley, Kenneth P., Pahud, Michel, Buxton, William Arthur Stewart, Pfeuffer, Ken
• Primary Examiner(s): Wills-Burns, Chineyere D
• Application Number:
  US15/437,387
• Publication Number:
  US 20180239482A1
• Time in Patent Office: 1,163 Days
• Field of Search: 345173
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04807 Pen manipulated menu
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06F 40/18 of spreadsheets form-filling G06F40/174

Abstract

Thumb+pen inputs are described herein, to improve the functionality of touch enabled devices for accepting bimanual input in situations where the device is gripped or supported by one of the user'"'"'s hands, leaving only one hand free. The thumb of an engaging hand is identified and controls are placed within its range of motion to enhance the functionality provided by the free hand. The actions of the thumb can be used to determine how pen actions made using the other hand are interpreted. Alternatively, the pen can indicate an object through pointing, while the thumb indirectly manipulates one or more of its parameters through touch controls. Marking menus, spring-loaded modes, indirect input, and conventional multi-touch interfaces are applied with respect to the bimanual input mode in which one hand is positioned to hold or support the device, and the other hand is free to improve device operability and accessibility.

Claims

1. A method for providing bimanual interactions for touchscreen enabled devices, comprising:

providing an application canvas accepting inputs from a touch interface of a user device;

detecting a contextual interaction of a gripping hand at an edge of the user device, wherein the gripping hand is a non-preferred hand of a user holding the user device;

in response to detecting the contextual interaction, displaying a marking menu comprised of thumb controls in the application canvas within a range of motion of a thumb of the gripping hand;

receiving, from the thumb of the gripping hand, a selection of a first thumb control of the marking menu, wherein the first thumb control is a spring-loaded mode control associated with a first mode;

while the first thumb control is being held by the thumb of the gripping hand, receiving a first set of actions made using a non-gripping hand;

interpreting the first set of actions based on the first mode;

applying the interpreted first set of actions to the application canvas;

in response to receiving a drag motion from the thumb of the gripping hand, re-displaying the marking menu comprised of the thumb controls in the application canvas;

receiving, from the thumb of the gripping hand, a selection of a second thumb control of the marking menu, wherein the second thumb control is a spring-loaded mode control associated with a second mode;

while the second thumb control is being held by the thumb of the gripping hand, receiving a second set of actions made using the non-gripping hand;

interpreting the second set of actions made using the non-gripping hand based on the second mode; and

applying the interpreted second set of actions to the application canvas.

2. The method of claim 1, further comprising:

while no thumb control is being held by the thumb of the gripping hand and a third set of actions made using the non-gripping hand is received, interpreting the third set of actions based on a default mode, wherein the default mode is distinct from the first mode and

the second mode.

3. The method of claim 1, further comprising:

determining the gripping hand is the non-preferred hand of the user holding the user device.

4. The method of claim 1, wherein displaying the marking menu comprised of the thumb controls in the application canvas further comprises:

determining a hand size of the gripping hand; and

displaying the thumb controls in the application canvas based on the hand size.

5. The method of claim 1, wherein displaying the marking menu comprised of the thumb controls in the application canvas further comprises:

determining a context of the gripping hand; and

positioning the thumb controls at a location suitable on the application canvas for interaction with the thumb based on the context.

6. A system for providing a unified system for bimanual interactions on flexible representations of content, comprising:

a processor; and

a memory including instructions, which when executed by the processor, causes the system to;

detect a contextual interaction of a gripping hand at an edge of a user device, the user device including a touch interface and providing an application canvas accepting inputs from the touch interface, wherein the gripping hand is a non-preferred hand of a user holding the user device;

in response to detecting the contextual interaction, display a marking menu comprised of thumb controls in the application canvas based on the contextual interaction within a range of motion of a thumb of the gripping hand for actuation by the thumb of the gripping hand;

receive, from the thumb of the gripping hand, a selection of a first thumb control of the marking menu, wherein the first thumb control is a spring-loaded mode control associated with a first mode;

while the first thumb control is being held by the thumb of the gripping hand, receive a first set of actions made by a pen held by a non-gripping hand;

interpret the first set of actions based on the first mode;

apply the interpreted first set of actions to the application canvas;

in response to receiving a drag motion from the thumb of the gripping hand, re-display the marking menu comprised of the thumb controls in the application canvas;

receive, from the thumb of the gripping hand, a selection of a second thumb control of the marking menu, wherein the second thumb control is a spring-loaded mode control associated with a second mode;

while the second thumb control is being held by the thumb of the gripping hand, receive a second set of actions made by the pen held by the non-gripping hand;

interpret the second set of actions based on the second mode; and

apply the interpreted second set of actions to the application canvas.

7. The system of claim 6, wherein the processor further causes the system to:

while no thumb control is being held by the thumb of the gripping hand and a third set of actions made by the pen held by the non-gripping hand is received, interpret the third set of actions based on a default mode, wherein the default mode is distinct from the first mode and

the second mode.

8. The system of claim 6, wherein the processor further causes the system to:

receive a selection of an object in the application canvas via the first set of actions; and

modify the object according to the interpreted first set of actions.

9. The system of claim 6, the processor further causes the system to:

determine the gripping hand is the non-preferred hand of the user holding the user device.

10. The system of claim 6, wherein the processor further causes the system to:

determine a hand size of the gripping hand; and

display the thumb controls in the application canvas based on the hand size.

11. The system of claim 6, wherein the processor further causes the system to:

determine a context in the application canvas; and

select the thumb controls for display in the application canvas based on the context.

12. The system of claim 11, wherein the context affects a location and a size of display for the thumb controls.

13. The system of claim 11, further comprising a motion sensor and an orientation sensor, wherein the context is determined according to the motion sensor and the orientation sensor.

14. The system of claim 6, wherein the first set of actions and the second set of actions are received from a finger of the non-gripping hand or a device held by the non-gripping hand.

15. The system of claim 6, wherein the first set of actions and the second set of actions include one or more of:

contact with the user device and detected hover inputs in proximity with the user device.

16. A computer readable storage media including computer readable instructions, which when executed by a processor, performs steps for providing bimanual interactions for touchscreen enabled devices, comprising:

detecting a contextual interaction of a gripping hand at an edge of a user device, the user device including a touch interface and providing an application canvas accepting inputs from the touch interface, wherein the gripping hand is a non-preferred hand of a user holding the user device;

in response to detecting the contextual interaction, displaying a marking menu comprised of thumb controls within a range of motion of a thumb of the gripping hand in the application canvas based on the contextual interaction for actuation by the thumb of the gripping hand including;

determining a hand size of the gripping hand; and

displaying the thumb controls in the application canvas based on the hand size;

receiving, from the thumb of the gripping hand, a selection of a first thumb control of the marking menu, wherein the first thumb control is a spring-loaded mode control associated with a first mode;

while the first thumb control is being held by the thumb of the gripping hand, receiving a first set of actions made by a pen held by a non-gripping hand;

interpreting the first set of actions based on the first mode;

applying the interpreted first set of actions to the application canvas;

in response to receiving a drag motion from the thumb of the gripping hand, re-displaying the marking menu comprised of the thumb controls in the application canvas;

receiving, from the thumb of the gripping hand, a selection of a second thumb control of the marking menu, wherein the second thumb control is a spring-loaded mode control associated with a second mode;

while the second thumb control is being held by the thumb of the gripping hand, receiving a second set of actions made by the pen held by the non-gripping hand;

interpreting the second set of actions based on the second mode; and

applying the interpreted second set of actions to the application canvas.

17. The computer readable storage media of claim 16, further comprising:

while no thumb control is being held by the thumb of the gripping hand and a third set of actions made by the pen held by the non-gripping hand is received, interpreting the third set of actions based on a default mode, wherein the default mode is distinct from the first mode; and

the second mode.