Multi-device multi-user sensor correlation for pen and computing device interaction

• US 9,870,083 B2
• Filed: 06/12/2014
• Issued: 01/16/2018
• Est. Priority Date: 06/12/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hinckley, Ken, Benko, Hrvoje, Pahud, Michel, Wilson, Andrew D., Irani, Pourang Polad, Guimbretiere, Francois
• Primary Examiner(s): McLoone, Peter D
• Application Number:
  US14/303,234
• Publication Number:
  US 20150363034A1
• Time in Patent Office: 1,314 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04162 for exchanging data with external devices, e.g. smart pens, via the digitiser sensing hardware
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A grip of a primary user on a touch-sensitive computing device and a grip of a secondary user on the touch-sensitive computing device are sensed and correlated to determine whether the primary user is sharing or handing off the computing device to the secondary user. In the case of handoff, capabilities of the computing device may be restricted, while in a sharing mode only certain content on the computing device is shared. In some implementations both a touch-sensitive pen and the touch-sensitive computing device are passed from a primary user to a secondary user. Sensor inputs representing the grips of the users on both the pen and the touch-sensitive computing device are correlated to determine the context of the grips and to initiate a context-appropriate command in an application executing on the touch-sensitive pen or the touch-sensitive computing device. Meta data is also derived from the correlated sensor inputs.

Claims

1. A computer-implemented process for initiating user interface actions, comprising using a computer for:

receiving sensor signals known to represent a grip of a primary user on a touch-sensitive computing device;

receiving sensor signals known to represent a grip of a secondary user on the touch-sensitive computing device at substantially the same time the signal for the grip of the primary user is received;

correlating the sensor signals for the grips of the primary and secondary users to determine that;

the primary user and the secondary user are concurrently sharing the touch-sensitive computing device while the grips of the primary and secondary users remain concurrent, and

the touch-sensitive computing device is being handed off from the primary user to the secondary user when the orientation of a plane of the touch-sensitive computing device is approximately level while the grips of the primary and secondary users remain concurrent, and then the orientation of the plane of the of the touch-sensitive device is angled towards the secondary user once the grip of the primary user is no longer detected; and

using the determination to initiate a command in an application executing on the touch-sensitive computing device.

2. The computer-implemented process of claim 1 wherein the determination is made that the touch-sensitive computing device is a device with a display screen.

3. The computer-implemented process of claim 2 wherein the orientation of the plane of the touch-sensitive device is angled towards a preferred viewing angle of the display screen for the secondary user once the grip of the primary user is no longer detected.

4. The computer-implemented process of claim 2 wherein one or more capabilities of the touch-sensitive computing device are restricted following the handoff.

5. The computer-implemented process of claim 1 wherein a context of at least one contact with the touch-sensitive device is determined by how the primary user or the secondary user grips the touch-sensitive device.

6. The computer-implemented process of claim 1, wherein each user is designated to have a different role with a different set of permissions in a user interface of the touch-sensitive computing device when the touch-sensitive device is operating in a sharing mode.

7. The computer-implemented process of claim 6, further comprising displaying one or more annotation layers on a display of the touch-sensitive computing device on which the secondary user is only allowed to view and edit content currently displayed on the touch-sensitive computing device.

8. The computer-implemented process of claim 7 wherein the secondary user is not allowed to navigate documents or files that are not displayed on the touch-sensitive computing device.

9. The computer-implemented process of claim 6 wherein the secondary user is not allowed to make digital copies of files on the touch-sensitive computing device.

10. The computer-implemented process of claim 1, wherein the touch-sensitive computing device is a touch-sensitive pen, further comprising:

transferring data from the primary user to the secondary user via the touch-sensitive pen; and

the secondary user downloading the data to another touch-sensitive computing device via the touch-sensitive pen.

11. A computer-implemented process for initiating user interface actions, comprising using a computer for:

concurrently sensing a first grip of a primary user on a touch-sensitive pen and a second grip of the primary user on a touch-sensitive computing device;

concurrently with sensed first and second grips of the primary user on the touch-sensitive pen and the touch-sensitive computing device, sensing a third grip of a secondary user on the touch-sensitive pen and a fourth grip on the touch-sensitive computing device;

correlating signals received from the concurrently sensed grips of the primary and secondary users on the touch-sensitive pen and the touch-sensitive computing device to determine a context of the grips; and

using the context of the grips to initiate a command in an application executing on the touch-sensitive pen or the touch-sensitive computing device.

12. The computer-implemented process of claim 11 further comprising classifying the first grip of the primary user and the third grip of the secondary user on the touch-sensitive pen into a category of a writing grip, tuck grip, palm grip, and no grip.

13. The computer-implemented process of claim 12 wherein the grips of the primary and the secondary user represent a handoff of both the pen and the touch-sensitive computing device to the secondary user from the primary user.

14. The computer-implemented process of claim 13 wherein the handoff is detected by determining that the orientations of a plane of the touch-sensitive pen and a plane of the touch-sensitive computing device each are level when the grips on the touch-sensitive pen and the touch-sensitive computing device are sensed.

15. The computer-implemented process of claim 14 wherein one or more capabilities of the touch-sensitive pen or the touch-sensitive computing device are restricted following the handoff.

16. The computer-implemented process of claim 13 wherein the handoff is determined by the grips of the primary and secondary user and that the touch-sensitive pen is resting on top of the touch-sensitive computing device at the handoff.

17. A system for finding meta information, comprising:

a computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to;

concurrently sense a first grip of a primary user on a first touch-sensitive device and a second grip of the primary user on a second touch-sensitive device;

concurrently with the sensed first and second grips of the primary user on the first touch-sensitive device and the second touch-sensitive device, sense a third grip of a secondary user on the first touch-sensitive device and a fourth grip of the secondary user on the second touch-sensitive device;

correlate signals generated by the simultaneous contacts from the users on the touch-sensitive devices;

determine a context of the simultaneous contacts from the users on the touch-sensitive devices based on the correlation of the signals;

for each determined context, of a simultaneous contact, label the determined context of the contact as metadata for use in an application.

18. The system of claim 17 wherein at least one touch-sensitive device is a touch-sensitive pen and at least one touch-sensitive device is a touch-sensitive computing device that are being simultaneously used by the same user.

19. The system of claim 18 wherein the determined context of at least one contact is derived by how the user grips the touch-sensitive pen.

20. The system of claim 17 wherein the derived context of the contact is derived by a signal representing a galvanic skin response.