THUMB AND PEN INTERACTION ON A MOBILE DEVICE

• US 20180239482A1
• Filed: 02/20/2017
• Published: 08/23/2018
• Est. Priority Date: 02/20/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hinckley, Kenneth P., Pahud, Michel, Buxton, William Arthur Stewart, Pfeuffer, Ken
• Granted Patent: US 10,635,291 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04807 Pen manipulated menu
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06F 40/18 of spreadsheets form-filling G06F40/174

Abstract

Thumb+pen inputs are described herein, to improve the functionality of touch enabled devices for accepting bimanual input in situations where the device is gripped or supported by one of the user'"'"'s hands, leaving only one hand free. The thumb of an engaging hand is identified and controls are placed within its range of motion to enhance the functionality provided by the free hand. The actions of the thumb can be used to determine how pen actions made using the other hand are interpreted. Alternatively, the pen can indicate an object through pointing, while the thumb indirectly manipulates one or more of its parameters through touch controls. Marking menus, spring-loaded modes, indirect input, and conventional multi-touch interfaces are applied with respect to the bimanual input mode in which one hand is positioned to hold or support the device, and the other hand is free to improve device operability and accessibility.

Claims

1. A method for providing bimanual interactions for touchscreen enabled devices, comprising:

providing an application canvas accepting touch inputs from the touch interface;

receiving a selection of a thumb control of the thumb controls;

receiving an input in the application canvas;

identifying an action based on the selected thumb control and the input; and

applying the action to the application canvas.

2. The method of claim 1, wherein applying the action to the application canvas further comprises:

applying a first mode of the input within the application canvas while the thumb control input is being received; and

applying a second mode of the input within the application canvas while the thumb control input is not being received.

3. The method of claim 1, wherein applying the action to the application canvas further comprises:

applying the action based on the selected thumb control as modified by the input.

4. The method of claim 1, wherein applying the action to the application canvas further comprises:

selecting an input mode for the input according to the selected thumb control; and

applying the modified input within the application canvas.

5. The method of claim 1, further comprising:

prior to displaying the thumb controls in the application canvas, determining whether the gripping hand is preferred or non-preferred for a user;

in response to determining that the gripping hand is preferred, not displaying the thumb controls in the application canvas; and

in response to determining that the gripping hand is non-preferred, displaying the thumb controls in the application canvas.

6. The method of claim 1, wherein displaying the thumb controls in the application canvas further comprises:

determining a hand size of the gripping hand; and

displaying the thumb controls in the application canvas based on the hand size.

7. The method of claim 1, wherein displaying the thumb controls in the application canvas further comprises:

determining a context of a gripping hand; and

positioning the thumb controls at a location suitable on the display for interaction with the thumb based on the context.

8. A system for providing a unified system for bimanual interactions on flexible representations of content, comprising:

a processing unit; and

a memory including instructions, which when executed by the processing unit, causes the system to;

detect a contextual interaction of a gripping hand on a user device, the user device including a touch interface and providing an application canvas accepting touch inputs from the touch interface;

in response to detecting the contextual interaction, display thumb controls in the application canvas based on the contextual interaction for actuation by a thumb of the gripping hand;

receive a selection of a given thumb control of the thumb controls;

receive a pen input in the application canvas;

identify an action based on the selected thumb control and the input; and

apply the given action to the application canvas.

9. The system of claim 8, wherein the processing unit further causes the system to:

apply a first mode of the pen input within the application canvas while the thumb control input is being received; and

apply a second mode of the pen input within the application canvas while the thumb control input is not being received.

10. The system of claim 8, wherein the processing unit further causes the system to:

receive a selection of an object in the application canvas via the pen input; and

modify the object according to the given action.

11. The system of claim 8, wherein the processing unit further causes the system to:

select an input mode for the pen input according to the given action; and

apply the modified pen input within the application canvas.

12. The system of claim 8, the processing unit further causes the system to:

prior to displaying the thumb controls in the application canvas, determine whether the gripping hand is preferred or non-preferred for a user;

in response to determining that the gripping hand is non-preferred, display primary thumb controls in the application canvas as the thumb controls; and

in response to determining that the gripping hand is preferred, display secondary thumb controls in the application canvas as the thumb controls, wherein the secondary thumb controls include different controls from the primary thumb controls or no controls.

13. The system of claim 8, wherein the processing unit further causes the system to:

determine a hand size of the gripping hand; and

display the thumb controls in the application canvas based on the hand size.

14. The system of claim 8, wherein the processing unit further causes the system to:

determine a context in the application canvas; and

select the thumb controls for display in the application canvas based on the context.

15. The system of claim 14, wherein the context affects a location and a size of display for the thumb controls.

16. The system of claim 14, further comprising a motion sensor and an orientation sensor, wherein the context is determined according to the motion sensor and the orientation sensor.

17. The system of claim 8, wherein the pen input is received from a non-gripping, including a finger of the non-gripping hand or a device held by the non-gripping hand.

18. The system of claim 8, wherein the pen input includes:

contact with the user device and detected hover inputs in proximity with the user device.

20. The computer readable storage media of claim 15, wherein applying the given action in conjunction with the pen input further comprises:

applying a first mode of the pen input within the application canvas while the thumb control input is being received; and

applying a second mode of the pen input within the application canvas while the thumb control input is not being received.

19. A computer readable storage media including computer readable instructions, which when executed by a processing unit, performs steps for providing bimanual interactions for touchscreen enabled devices, comprising:

detecting a grip interaction of a gripping hand on a user device, the user device including a touch interface and providing an application canvas accepting touch inputs from the touch interface;

in response to detecting the grip interaction, displaying thumb controls in the application canvas based on the grip interaction for actuation by a thumb of the gripping hand including;

determining a hand size of the gripping hand; and

displaying the thumb controls in the application canvas based on the hand size;

receiving a selection of a given thumb control of the thumb controls associated with a given action;

receiving a pen input in the application canvas; and

applying the given action in conjunction with the pen input.