Motion and context sharing for pen-based computing inputs

• US 9,201,520 B2
• Filed: 05/28/2013
• Issued: 12/01/2015
• Est. Priority Date: 02/11/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Benko, Hrvoje, Chen, Xiang, Hinckley, Kenneth Paul
• Primary Examiner(s): Zhou, Hong
• Application Number:
  US13/903,944
• Publication Number:
  US 20130257777A1
• Time in Patent Office: 917 Days
• Field of Search: 345173-179, 178 1801- 2001, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device

Abstract

A “Motion and Context Sharing Technique” uses a pen or stylus enhanced to incorporate multiple sensors, i.e., a “sensor pen,” and a power supply to enable various input techniques and gestures. Various combinations of pen stroke, pressure, motion, and other sensor pen inputs are used to enable various hybrid input techniques that incorporate simultaneous, concurrent, sequential, and/or interleaved, sensor pen inputs and touch inputs (i.e., finger, palm, hand, etc.) on displays or other touch sensitive surfaces. This enables a variety of motion-gesture inputs relating to the context of how the sensor pen is used or held, even when the pen is not in contact or within sensing range of the computing device digitizer. In other words, any particular touch inputs or combinations of touch inputs are correlated with any desired sensor pen inputs, with those correlated inputs then being used to initiate any desired action by the computing device.

Claims

1. A computer-implemented process, comprising using a computer to perform process actions for:

receiving sensor inputs from a plurality of sensors coupled to a stylus device while the stylus device is outside a hover range of a touch-sensitive computing device, said sensor inputs representing one or more intentional user interactions with the stylus device;

detecting one or more finger touches on any touch sensitive surface of the touch-sensitive computing device, said finger touches occurring while the intentional user interactions with the stylus device are occurring;

evaluating the sensor inputs in combination with the finger touches to identify an intended motion gesture from a plurality of available motion gestures; and

initiating a user interface action corresponding to the intended motion gesture in an application executing on the touch-sensitive computing device.

2. The computer-implemented process of claim 1 wherein one or more of the sensors are used to determine intentional motion of the stylus device relative to concurrent intentional motion of the touch-sensitive computing device, and wherein those intentional motions are correlated for use in identifying the intended motion gesture.

3. The computer-implemented process of claim 1 further comprising initiating an alert on the touch-sensitive computing device when one or more sensors coupled to the touch-sensitive computing device indicate that the touch-sensitive computing device is moving away from the stylus device.

4. The computer-implemented process of claim 1 wherein one or more of the sensors are used to determine orientation of the stylus device relative to the touch-sensitive computing device, and wherein that relative orientation is used to identify the intended motion gesture.

5. The computer-implemented process of claim 1 wherein the finger touches include movement of one or more fingers across any touch sensitive surface of the touch-sensitive computing device.

6. The computer-implemented process of claim 1 further comprising a user interface for specifying sensor inputs and touches corresponding to one or more of the plurality of available motion gestures.

7. The computer-implemented process of claim 1 wherein one or more of the sensors indicate a rotation of the stylus device around a long axis of the stylus device concurrently with a touch of an object displayed on a touch screen of the touch-sensitive computing device, and wherein the intended motion gesture corresponds to a user interface action for rotating the touched object.

8. The computer-implemented process of claim 1 wherein one or more of the sensors indicate that the stylus device is in an approximately perpendicular orientation relative to a touch screen of the touch-sensitive computing device, and wherein the intended motion gesture corresponds to a user interface action for activating a marking menu on the display device.

9. The computer-implemented process of claim 1 wherein one or more of the sensors indicate a rotation of the stylus device around a long axis of the stylus device, and wherein the intended motion gesture corresponds to a user interface action for undoing a prior action of the application executing on the touch-sensitive computing device.

10. The computer-implemented process of claim 1 wherein one or more of the sensors are used to determine when the stylus device is picked up from a surface, and wherein the intended motion gesture corresponds to a user interface action for presenting a context sensitive menu relating to a currently active object in the application executing on the touch-sensitive computing device.

11. The computer-implemented process of claim 1 wherein one or more of the sensors are used to determine motion of the stylus device and motion of a touch on the touch-sensitive computing device, and when the those motions are in generally the same direction, using the directional correlation of those motions to identify the intended motion gesture.

12. A method for initiating actions in an application executing on a touch-sensitive computing device, comprising:

receiving sensor inputs from a plurality of sensors coupled to a stylus device while the stylus device is outside a hover range of a touch-sensitive computing device, said sensor inputs representing one or more intentional user interactions with the stylus device;

detecting one or more finger touches on any touch sensitive surface of the touch-sensitive computing device, said finger touches occurring while the intentional user interactions with the stylus device are occurring;

evaluating the sensor inputs in combination with any finger touches to identify an intended motion gesture from a plurality of available motion gestures; and

initiating a user interface action corresponding to the intended motion gesture in an application executing on the touch-sensitive computing device.

13. The method of claim 12 wherein one or more of the sensors indicate tapping on a barrel of the stylus device, and wherein the intended motion gesture corresponds to a user interface action for emulating a button press of an input device.

14. The method of claim 12 wherein one or more of the sensors indicate tapping on a barrel of the stylus device, and wherein the intended motion gesture corresponds to a user interface action for redoing a prior action of the application executing on the touch-sensitive computing device.

15. The method of claim 12 wherein one or more of the sensors indicate a tilting of the stylus device relative to a display of the touch-sensitive computing device, and wherein a finger touch on the touch-sensitive computing device corresponds to two or more layers objects, and wherein the intended motion gesture corresponds to a user interface action for cycling between the layers.

16. The method of claim 12 wherein the application executing on the touch-sensitive computing device is a painting type application and wherein one or more of the sensors indicate rapping of the stylus device against a surface while there is a touch on a display of the touch-sensitive computing device, and wherein the intended motion gesture corresponds to a user interface action for digitally spattering a plurality of digital paint drops around the touch on the display.

17. A computer storage media having computer executable instructions stored therein, said instructions causing a computing device to execute a method comprising:

receiving sensor inputs from a plurality of sensors coupled to a stylus device while the stylus device is outside a hover range of a touch-sensitive computing device, said sensor inputs representing one or more intentional user interactions with the stylus device;

detecting one or more finger touches on any touch sensitive surface of the touch-sensitive computing device, said finger touches occurring at the same time as the sensor inputs received from the stylus device;

evaluating the sensor inputs in combination with any finger touches to identify an intended motion gesture from a plurality of available motion gestures; and

initiating a user interface action corresponding to the intended motion gesture in an application executing on the touch-sensitive computing device.

18. The computer storage media of claim 17 wherein one or more of the sensors are used to determine motion of the stylus device relative to motion of the touch-sensitive computing device, and wherein those motions are correlated for use in identifying the intended motion gesture.

19. The computer storage media of claim 17 further comprising initiating an alert on the touch-sensitive computing device when one or more sensors coupled to the touch-sensitive computing device indicate that the touch-sensitive computing device is moving away from the stylus device.

20. The computer storage media of claim 17 wherein one or more of the sensors are used to determine orientation of the stylus device relative to the touch-sensitive computing device, and wherein that relative orientation is used to identify the intended motion gesture.