Touch event anticipation in a computing device

• US 9,354,804 B2
• Filed: 12/29/2010
• Issued: 05/31/2016
• Est. Priority Date: 12/29/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Berkes, Otto, Geiger, Avi, Amdur, Meredith E.
• Primary Examiner(s): Tung, David
• Application Number:
  US12/981,184
• Publication Number:
  US 20120169646A1
• Time in Patent Office: 1,980 Days
• Field of Search: 345156-184
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Systems and methods for anticipation of touch events in a computing device are provided. The computing device may include a multi-touch sensitive display including a sensor configured to sense a position and/or movement of a hand. A graphical user interface (GUI) may be displayed, having a state including a plurality of possible touch inputs. The computing device may further include a touch event prediction module configured to compute one or more anticipated touch inputs based on the sensed hand position and/or movement, and the state of the GUI with the plurality of possible user inputs. The computing device may further include a preprocessing module configured to preprocess data for each anticipated touch input, and upon the detection of an actual touch input received from the user that matches one of the anticipated touch inputs, displaying the preprocessed data for the actual touch input on the GUI.

Claims

1. A computing device, comprising:

a multi-touch sensitive display including a sensor configured to first sense spaced apart input indicating a position and/or movement of a user hand when the user hand is spaced apart a distance from a top surface of the display, the sensor being further configured to then sense an actual touch input when the user hand is contacting the display;

a graphical user interface displayed on the display, the graphical user interface having a state including a plurality of possible touch inputs, each of the plurality of touch inputs associated with a specific element of the graphical user interface;

a touch event prediction module, executed by a processor, configured to compute one or more anticipated touch inputs based on the first sensed spaced apart input indicating the user hand position and/or movement relative to a current state of the user interface with the plurality of possible touch inputs prior to detecting the actual touch input from the user, the touch event prediction module including a statistics engine configured to compute an estimated probability that each of the anticipated touch inputs is selected based on prior usage data and based on at least two hover positions indicating movement of the user hand, and to instruct the preprocessing module to allocate preprocessing resources to one or more anticipated touch inputs with higher probability of selection and to forgo preprocessing of the other anticipated touch inputs; and

a preprocessing module, executed by the processor, configured to, at the computing device, generate preprocessed data for each anticipated touch input, the preprocessed data associated with each of one or more elements of the graphical user interface, and upon the subsequent detection of the actual touch input received from the user which matches one of the anticipated touch inputs, cause the preprocessed data for the actual touch input to be displayed on the graphical user interface, the preprocessing module being further configured to preload data to be displayed upon anticipation of a touch input into a cache, and/or perform an advance computation for the anticipated touch input.

2. The computing device of claim 1, wherein the sensor is an in-pixel optical sensor, capacitive touch sensor, resistive touch sensor, pressure sensor, or infrared sensor.

3. The computing device of claim 1, wherein the touch event prediction module and the preprocessing module are part of an application programming interface linking programs to operating system functionality on the computing device.

4. The computing device of claim 1, wherein the computing device is configured as a mobile telephone, a tablet computing device, a laptop computer, a desktop computer, a tabletop computer, or a kiosk computer.

5. The computing device of claim 1, wherein the sensor is configured to detect position and/or movement of one or more digits of the user hand and/or a stylus held by the user hand.

6. The computing device of claim 1, wherein the computing device is configured to display an indication to the user of the anticipated touch input, wherein the indication includes changing portions of the graphical user interface to become brighter, change color, fade and/or disappear to provide user feedback of the anticipated touch input.

7. The computing device of claim 1, wherein:

the sensor is configured to sense the spaced apart input by sensing that a shadow cast by the user hand onto the top surface of the display has become smaller and sharper over time; and

in response, the touch event prediction module is further configured to determine that the user hand is moving toward one of the elements of the graphical user interface.

8. A computing method, comprising:

displaying a graphical user interface including a state having a plurality of possible touch inputs on a touch sensitive display of a computing device, each of the plurality of touch inputs associated with a specific element of the graphical user interface;

sensing spaced apart input indicating a position and/or movement of a user hand via a sensor of the computing device, when the user hand is spaced apart an orthogonal distance from a top surface of the display;

computing one or more anticipated touch inputs based on the sensed spaced apart input indicating the user hand position and/or movement relative to a current state of the user interface with the plurality of possible touch inputs prior to detecting an actual touch input from a user when the user hand contacts the display;

computing an estimated probability that each of the anticipated touch inputs will be selected based on prior usage data and based on at least two hover positions indicating movement of the user hand;

preprocessing data at the computing device for anticipated touch inputs with a higher probability, and forgoing preprocessing for anticipated touch inputs with a lower probability by preloading data to be displayed upon performance of each anticipated touch input with the higher probability into a cache and performing an advanced computation for each touch input with the higher probability, the data associated with each of one or more elements of the graphical user interface;

detecting the actual touch input which matches one of the anticipated touch inputs when the user hand contacts the display; and

displaying the preprocessed data on the graphical user interface for the actual touch input.

9. The method of claim 8, wherein computing one or more anticipated touch inputs and preprocessing data for each anticipated touch input are performed at least in part by an application programming interface linking programs to operating system functionality on the computing device.

10. The method of claim 8, wherein sensing is performed by a sensor selected from the group consisting of in-pixel optical sensor, capacitive touch sensor, resistive touch sensor, and infrared sensor.

11. The method of claim 8, wherein sensing the position and/or movement of the user hand further includes:

detecting a position and/or movement of one or more digits of the user hand or a stylus held by the user hand.

12. The method of claim 8, further comprising:

displaying an indication to the user of the anticipated touch input on the display, wherein the indication includes changing portions of the graphical user interface to become brighter, change color, fade and/or disappear to provide user feedback of the anticipated touch input.

13. The computing device of claim 8, wherein the computing device is configured as a mobile telephone, a tablet computing device, a laptop computer, a desktop computer, a tabletop computer, or a kiosk computer.

14. A computing method, comprising:

displaying a graphical user interface on a touch sensitive display of a computing device, the graphical user interface including a state having a plurality of possible touch inputs, each of the plurality of touch inputs associated with a specific element of the graphical user interface;

sensing spaced apart input indicating a position and/or movement of a user hand when the user hand is spaced apart a distance from a top surface of the display via a sensor of the computing device;

computing one or more anticipated touch inputs based on the sensed spaced apart input indicating the user hand position and/or movement relative to a current state of the user interface with the plurality of possible touch inputs prior to detecting an actual touch input from a user when the user hand contacts the display; and

computing an estimated probability that each of the anticipated touch inputs will be selected based on prior usage data and based on at least two hover positions indicating movement of the user hand;

at the computing device, preprocessing data for anticipated touch inputs with a higher probability, and forgoing preprocessing for anticipated touch inputs with a lower probability by preloading data to be displayed upon performance of each anticipated touch input with the higher probability into a cache and/or by performing an advance computation for each anticipated touch input with the higher probability, the preloaded data associated with each of one or more elements of the graphical user interface;

detecting the actual touch input which matches one of the anticipated touch inputs when the user hand contacts the display; and

displaying the preprocessed data on the graphical user interface for the actual touch input;

wherein the computing device is configured as a mobile telephone, a tablet computing device, a laptop computer, a desktop computer, a tabletop computer, or a kiosk computer.