Sensor correlation for pen and touch-sensitive computing device interaction

• US 9,727,161 B2
• Filed: 06/12/2014
• Issued: 08/08/2017
• Est. Priority Date: 06/12/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Irani, Pourang Polad, Guimbretiere, Francois, Hinckley, Ken, Benko, Hrvoje, Pahud, Michel, Wilson, Andrew D.
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US14/303,203
• Publication Number:
  US 20150363035A1
• Time in Patent Office: 1,153 Days
• Field of Search: 178 1801- 1907, 345173-183
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04162 for exchanging data with external devices, e.g. smart pens, via the digitiser sensing hardware
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Pen and computing device sensor correlation technique embodiments correlate sensor signals received from various grips on a touch-sensitive pen and touches to a touch-sensitive computing device in order to determine the context of such grips and touches and to issue context-appropriate commands to the touch-sensitive pen or the touch-sensitive computing device. A combination of concurrent sensor inputs received from both a touch-sensitive pen and a touch-sensitive computing device are correlated. How the touch-sensitive pen and the touch-sensitive computing device are touched or gripped are used to determine the context of their use and the user'"'"'s intent. A context-appropriate user interface action based can then be initiated. Also the context can be used to label metadata.

Claims

1. A computer-implemented process for initiating user interface actions, comprising using a computer for:

receiving a combination of concurrent sensor inputs received from a touch-sensitive pen and a touch-sensitive computing device, comprising;

one or more sensor inputs based on grip patterns on the touch-sensitive pen,

one or more sensor inputs based touch patterns on the touch-sensitive computing device;

determining the user'"'"'s preferred hand and non-preferred hand from the sensor inputs, wherein the user'"'"'s preferred hand is determined by;

sensing a bump in a motion of the touch-sensitive pen when a contact is made with the touch-sensitive computing device; and

correlating the bump in a touch-sensitive pen'"'"'s motion with a motion of the touch-sensitive computing device to determine that the hand holding the touch-sensitive pen is the preferred hand;

using the sensor inputs to determine how the touch-sensitive pen is being gripped concurrent with how the touch-sensitive computing device is being touched; and

initiating a user interface action based on a combination of the grip pattern on the touch-sensitive pen device and the touch pattern on the touch-sensitive computing device.

2. The computer-implemented process of claim 1, further comprising using the determination of the user'"'"'s preferred hand as metadata.

3. The computer-implemented process of claim 1 further comprising using the signals representing the orientation of the touch-sensitive pen in determining the grip patterns.

4. The computer-implemented process of claim 1 further comprising:

sensing motion of the touch-sensitive computing device;

correlating the sensed touch-sensitive computing device motion with the touch patterns on the touch-sensitive computing device; and

using the correlated motion and touch patterns on the touch-sensitive computing device to suppress accidental screen content rotation on a display of the touch-sensitive computing device.

5. The computer-implemented process of claim 1, further comprising:

using the grip patterns, determining whether the user is making a gesture with a bare hand or whether the user is making a gesture with a hand holding the touch-sensitive pen.

6. The computer-implemented process of claim 1, further comprising:

determining that the touch-sensitive pen is held in a stowed position in the user'"'"'s preferred hand;

detecting a contact from the user'"'"'s preferred hand on a display of the touch-sensitive computing device; and

displaying context-specific tools in response to the contact.

7. The computer-implemented process of claim 1, further comprising:

determining that the touch-sensitive pen is held in the user'"'"'s preferred hand in a grip indicating that the touch-sensitive pen is not in a position to write;

detecting a contact from the user'"'"'s non-preferred hand on a display of the touch-sensitive computing device; and

displaying context-specific tools in response to the contact.

8. The computer-implemented process of claim 1, further comprising:

determining that the touch-sensitive pen is held in a grip indicating that the touch-sensitive pen is in a position to write in the user'"'"'s preferred hand;

detecting a contact from the user'"'"'s non-preferred hand on a display of the touch-sensitive computing device; and

displaying context-specific tools in response to the contact.

9. The computer-implemented process of claim 1, further comprising:

determining that the touch-sensitive pen is held in a grip indicating that the touch-sensitive pen is in a position to write in the user'"'"'s preferred hand;

detecting a multi-finger touch gesture from the user'"'"'s non-preferred hand on the display of the touch-sensitive computing device; and

displaying context-specific tools in response to the multi-finger touch gesture.

10. The computer-implemented process of claim 1, further comprising:

determining that the touch-sensitive pen is held in a grip indicating that the touch-sensitive pen is not in a position to write in the user'"'"'s preferred hand;

detecting a two-finger touch gesture from the user'"'"'s preferred hand on the display of the touch-sensitive computing device; and

displaying a magnifier tool that magnifies content on a display of the touch-sensitive computing device in response to the two-finger touch gesture.

11. The computer-implemented process of claim 1, further comprising:

determining a touch on a touch-sensitive screen of the touch-sensitive computing device is made by a user'"'"'s thumb at the same time a new hand grip pattern on a corresponding back portion of the touch-sensitive computing device'"'"'s case is detected; and

designating that the touch made by the thumb on the touch-sensitive screen is unintentional.

12. The computer-implemented process of claim 1, further comprising:

determining which hand a user is holding the touch-sensitive computing device with; and

displaying a thumb menu in a location on the screen of the touch-sensitive computing device that can be activated by the user with the thumb of the hand holding the touch-sensitive computing device.

13. The computer-implemented process of claim 12 wherein the thumb menu fades away when the user no longer touches the touch-sensitive computing device.

14. A computer-implemented process for finding meta information, comprising:

using a computing device for;

correlating signals of simultaneous contacts on two or more touch-sensitive devices by a single user;

determining the user'"'"'s preferred hand from the correlated signals of the simultaneous sensor inputs, wherein the user'"'"'s preferred hand is determined by;

sensing a bump in a motion of the a first touch-sensitive device when a contact is made with the first touch-sensitive computing device; and

correlating the bump in the first touch-sensitive device'"'"'s motion with a motion of the second touch-sensitive device to determine that the hand holding the first touch-sensitive device is the preferred hand;

determining a context of the contacts on the two or more touch-sensitive devices based on the correlation of the signals;

labeling the determined context of the contact as metadata for use in an application.

15. The computer-implemented process of claim 14 wherein one touch-sensitive device is a touch-sensitive pen and the other touch-sensitive device is a touch-sensitive computing device, and wherein the context of the contacts is determined by how the touch-sensitive pen is being gripped and how the touch-sensitive computing device is being touched with a user'"'"'s bare hand.

16. The computer-implemented process of claim 14, further comprising using the meta data to initiate a command in an application.

17. The computer-implemented process of claim 14, wherein one or more of the signals represent a finger print of a user which is used to identify the user.

18. The computer-implemented process of claim 14, further comprising using historical signal data to improve the correlation of the signals.