Application programming interface for multi-touch input detection

• US 10,289,239 B2
• Filed: 12/12/2016
• Issued: 05/14/2019
• Est. Priority Date: 07/09/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Pahud, Michel, Buxton, William, Hinckley, Kenneth P., Webb, Andrew M., Ofek, Eyal
• Primary Examiner(s): Merkoulova, Olga V
• Assistant Examiner(s): Suteerawongsa, Jarurat
• Application Number:
  US15/376,436
• Publication Number:
  US 20170090666A1
• Time in Patent Office: 883 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A sensing device, such as a user-wearable device (UWD) worn by a user of a touchscreen, may provide kinematic data of the sensing device or UWD and/or identification data of the user to a processor that operates the touchscreen. Such data may allow the processor to perform a number of user-touchscreen interactions, such as displaying user-specific windows or menus, processing user-manipulation of displayed objects, and determining which hand of a user performs a touch event, just to name a few examples.

Claims

1. A system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving data representative of acceleration of a sensing device from the sensing device during a touch event;

generating an acceleration pulse from the data representative of the acceleration of the sensing device where the acceleration pulse represents acceleration as a function of time and has an amplitude and a width;

determining whether the amplitude of the acceleration pulse exceeds a first detection threshold and the width of the acceleration pulse is below a second detection threshold after the touch event has occurred;

where a determination is made that the amplitude of the acceleration pulse exceeds the first detection threshold and the width of the acceleration pulse is below a second detection threshold, determining that the sensing device is held or worn on an arm that performed the touch event;

where a determination is made that the amplitude of the acceleration pulse does not exceed the detection threshold and the width of the acceleration pulse is below a second detection threshold, determining that the sensing device is held or worn on an arm opposite to the arm that performed the touch event; and

providing a first set of instructions for responding to the touch event if the sensing device is held or worn on the arm that performed the touch event or providing a second set of instructions for responding to the touch event if the sensing device is held or worn on the arm opposite to the arm that performed the touch event.

2. The system of claim 1, further comprising a touchscreen, wherein the operations further comprise receiving the touch event on the touchscreen.

3. The system of claim 1;

the operations further comprising;

receiving an identification of the sensing device before or during the touch event;

searching for the identification of the sensing device in a look-up table;

determining an identification of a user based, at least in part, on the searching; and

modifying the first set of instructions or the second set of instructions based, at least in part, on the identification of the user.

4. The system of claim 3, wherein the first set of instructions for responding to the touch event is associated with a dominant hand of the user and the second set of instructions for responding to the touch event is associated with a non-dominant hand of the user.

5. The system of claim 3, wherein the operations further comprise:

receiving information from a memory or from the sensing device regarding whether the sensing device is held or worn on a dominant hand of the user or a non-dominant hand of the user; and

modifying the first set of instructions or the second set of instructions based, at least in part, on the information.

6. The system of claim 1, the operations further comprising:

based, at least in part, on the data representative of acceleration of the sensing device, determining whether one finger or more than one finger of a user produced the touch event.

7. A method comprising:

in response to a touch event on a display surface;

querying a user wearable device (UWD) that is within a threshold distance from the display surface for identification of the MD;

querying the UWD for acceleration data of the UWD as measured by the UWD during the touch event, wherein the acceleration data is representable as a pulse having a width and an amplitude;

receiving the acceleration data;

matching the identification of the UWD in a look-up table with a corresponding identity of a user;

determining, based at least in part on the look-up table, placement information regarding whether the UWD is worn on a dominant hand of the user or a non-dominant hand of the user;

determining, based at least in part on (i) whether the height of the pulse representing the acceleration data exceeds a first detection threshold and the width is below a second detection threshold after the touch event has occurred; and

(ii) the placement information, if the dominant hand of the user produced the touch event or if the non-dominant hand of the user produced the touch event; and

generating a reaction on the display surface to the touch event based, at least in part, on the identity of the user.

8. The method of claim 7, wherein the generating the reaction to the touch event is further based, at least in part, on whether the dominant hand of the user produced the touch event or the non-dominant hand of the user produced the touch event.

9. The method of claim 7, further comprising:

determining a shape of an acceleration pulse from the acceleration data, wherein the acceleration pulse represents a magnitude of an acceleration of the UWD as a function of time, and wherein the determining if the dominant hand of the user produced the touch event or if the non-dominant hand of the user produced the touch event is further based, at least in part, on a shape of the pulse.

10. The method of claim 7, further comprising:

storing at least a portion of the acceleration data and associating the stored acceleration data with the user, and accessing the stored acceleration data during a machine learning process for determining behavior of the user during additional touch events.

11. The method of claim 7, further comprising:

storing at least a portion of the acceleration data and associating the stored acceleration data with the user, storing additional acceleration data of a second UWD of a second user and associating the stored additional acceleration data with the second user; and

accessing the stored acceleration data and the stored additional acceleration data during a machine learning process to determine whether the user or the second user produced the touch event.

12. The method of claim 7, wherein the look-up table includes data representative of historical touch events performed by the user.

13. The method of claim 7, wherein the identification of the ID comprises a globally unique identifier (GUID).

14. The method of claim 7, further comprising:

in response to the touch event, querying the MD for acceleration data measured by e UWD during the touch event;

receiving the acceleration data; and

determining a magnitude of force of the touch event based, at least in part, on the acceleration data, wherein the generating the reaction to the touch event is further based, at least in part, on the magnitude of force of the touch event.

15. A system comprising:

a memory storage device storing computer-executable instructions; and

one or more hardware processors that, having executed the computer-executable instructions, configure a system to implement an application programming interface, the application programming interface comprising;

a query module to query, in response to a touch event, a user wearable device (UWD) for kinematic data of the UWD as measured by the UWD during the touch event, the kinematic data including acceleration data of the UWD, the acceleration data corresponding to an acceleration pulse representing acceleration as a function of time and having a width and an amplitude;

a receive module to receive the kinematic data;

a touch-identifying module to determine whether the UWD is held or worn on an arm that performed the touch event based, at least in part, on whether the amplitude of the acceleration pulse exceeds a first detection threshold and the width of the acceleration pulse is below a second detection threshold after the touch event has occurred; and

a display module to generate a displayed reaction to the touch event based, at least in part, on the determination performed by the touch-identifying module.

16. The system of claim 15, wherein:

the query module is configured to query the UWD for identification of the UWD; and

the application programming interface further comprises;

a user-identifying module to identify a user based, at least in part, on the identification of the UWD.

17. The system of claim 16, wherein the application programming interface further comprises:

a menu-generating module to generate a custom menu based, at least in part, on the identity of the user.

18. The system, of claim 16, wherein the user-identifying module is configured to access the Cloud to identify the user based, at least in part, on the identification of the UWD.

19. The system of claim 15, wherein the kinematic data includes velocity data and orientation data of the UWD.