Model-based touch event location adjustment

• US 9,927,917 B2
• Filed: 10/29/2015
• Issued: 03/27/2018
• Est. Priority Date: 10/29/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Gunawardana, Asela, Paek, Timothy S., Kulkarni, Vishwas, Rudchenko, Dmitry, Bentley, Devlin, Nelson, David
• Primary Examiner(s): EURICE, MICHAEL
• Application Number:
  US14/927,397
• Publication Number:
  US 20170123586A1
• Time in Patent Office: 880 Days
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

An electronic device includes a display including a touch sensor, a contextual sensor configured to provide data indicative of an operating condition of the electronic device, one or more memories in which touch input instructions, touch event mapping instructions, and a touch event mapping model are stored, and a processor coupled to the memory. The processor is configured through execution of the touch input instructions to obtain data indicative of an initial assessment of touch event position via the touch sensor. The touch event mapping model maps the initial assessment of touch event position to an adjusted touch event position as a function of the data indicative of the operating condition of the electronic device. The processor is further configured through execution of the touch event mapping instructions to determine the adjusted touch event position based on the touch event mapping model, the data indicative of the initial assessment of touch event position, and the data indicative of the operating condition.

Claims

1. An electronic device comprising:

a display comprising a touch sensor;

a contextual sensor configured to provide data indicative of an operating condition of the electronic device;

one or more memories in which touch input instructions, touch event mapping instructions, and a touch event mapping model are stored; and

a processor coupled to the memory, the processor configured through execution of the touch input instructions to obtain data indicative of an initial assessment of touch event position via the touch sensor;

wherein the touch event mapping model maps the initial assessment of touch event position to an adjusted touch event position as a function of the data indicative of the operating condition of the electronic device, and

wherein the processor is further configured through execution of the touch event mapping instructions to determine the adjusted touch event position based on the touch event mapping model, the data indicative of the initial assessment of touch event position, and the data indicative of the operating condition.

2. The electronic device of claim 1, wherein the contextual sensor comprises an inertial measurement sensor such that the operating condition is indicative of acceleration and/or orientation of the electronic device.

3. The electronic device of claim 1, wherein:

the touch sensor comprises an array of touch sensor elements; and

the data indicative of the initial assessment of touch event position comprises raw data for each touch sensor element of the array.

4. The electronic device of claim 1, wherein:

the data indicative of the initial assessment of touch event position specifies the initial assessment of touch event position;

the touch sensor comprises an array of touch sensor elements;

the processor is further configured through execution of the touch input instructions to obtain raw data for each touch sensor element of the array; and

the processor is further configured through execution of the touch event mapping instructions to determine the adjusted touch event position based on the raw data and the data indicative of the initial assessment of touch event position.

5. The electronic device of claim 1, wherein the touch event mapping model comprises a set of boosted regression trees.

6. The electronic device of claim 1, wherein the one or more memories comprise a non-volatile memory in which the touch event mapping model is stored.

7. The electronic device of claim 1, wherein the touch event mapping model maps the initial assessment of touch event position to the adjusted touch event position to correct for finger curvature.

8. The electronic device of claim 1, wherein the touch event mapping model is configured as a regression tree.

9. The electronic device of claim 8, wherein:

the touch sensor comprises an array of touch sensor elements; and

the regression tree comprises a node at which a respective branch of the regression tree is selected based on whether a respective touch sensor element of the array is activated.

10. The electronic device of claim 1, wherein:

touch event smoothing instructions are stored in the one or more memories;

the processor is configured through execution of the touch input instructions to obtain a series of past observed touch event positions; and

the processor is configured through execution of the touch event smoothing instructions to determine a state of a dynamic system model based on the series of past observed touch event positions, the state being indicative of a predicted touch event position, and to provide a smoothed touch event position as a function of the predicted touch event position and a current observed touch event position.

11. The electronic device of claim 10, wherein the processor is configured through execution of the touch event smoothing instructions to use the adjusted touch event position as the current observed touch event position.

12. An electronic device comprising:

a display comprising a touch sensor;

an inertial measurement sensor configured to provide inertial measurement data for the electronic device;

one or more memories in which instructions and a touch event mapping model are stored; and

a processor coupled to the memory, the processor configured through execution of the instructions to obtain an initial assessment of touch event position via the touch sensor;

wherein the touch event mapping model maps the initial assessment of touch event position to an adjusted touch event position as a function of the data indicative of the operating condition of the electronic device, and

wherein the processor is further configured through execution of the instructions to provide the adjusted touch event position based on the touch event model and the initial touch event position.

13. The electronic device of claim 12, wherein:

the touch event mapping model is configured as a regression tree;

the touch sensor comprises an array of touch sensor elements; and

the regression tree comprises a node at which a respective branch of the regression tree is selected based on whether a respective touch sensor element of the array is activated.

14. The electronic device of claim 12, wherein:

touch event smoothing instructions are stored in the one or more memories;

the processor is configured through execution of the touch input instructions to obtain a series of past observed touch event positions;

the processor is configured through execution of the touch event smoothing instructions to determine a state of a dynamic system model based on the series of past observed touch event positions, the state being indicative of a predicted touch event position, and to provide a smoothed touch event position as a function of the predicted touch event position and a current observed touch event position; and

the processor is configured through execution of the touch event smoothing instructions to use the adjusted touch event position as the current observed touch event position.

15. The electronic device of claim 12, wherein:

the touch sensor comprises an array of touch sensor elements; and

the processor is further configured through execution of the instructions to obtain raw data for each touch sensor element of the array.

16. The electronic device of claim 15, wherein the processor is further configured through execution of the instructions to determine the adjusted touch event position based on the raw data and the initial assessment of touch event position.

17. An electronic device comprising:

a display comprising a touch sensor;

one or more memories in which touch input instructions and model-based touch event adjustment instructions are stored; and

a processor coupled to the memory, the processor configured through execution of the touch input instructions to obtain, via the touch sensor, a series of past observed touch event positions and a current observed touch event position;

wherein the processor is further configured through execution of the model-based touch event adjustment instructions to determine a state of a dynamic system model based on the series of past observed touch event positions, the state being indicative of a predicted touch event position, and to provide an adjusted touch event position as a function of the predicted touch event position and the current observed touch event position.

18. The electronic device of claim 17, wherein the state of the dynamic system model is further indicative of a predicted direction of movement for the series of past observed touch event positions and a predicted speed of movement for the series of past observed touch event positions.

19. The electronic device of claim 17, wherein the dynamic system model comprises a Kalman filter or a Hidden Markov model.

20. The electronic device of claim 17, wherein the function comprises an interpolation between the predicted touch event position and the current observed touch event position.

21. The electronic device of claim 20, wherein:

the state of the dynamic system model comprises an uncertainty parameter; and

the interpolation is weighted in accordance with the uncertainty parameter.

22. An electronic device comprising:

a display comprising a touch sensor;

one or more memories in which touch input instructions, touch event mapping instructions, and a touch event mapping model are stored; and

a processor coupled to the memory, the processor configured through execution of the touch input instructions to obtain data indicative of an initial assessment of touch event position via the touch sensor;

wherein the touch event mapping model maps the initial assessment of touch event position to an adjusted touch event position to correct for finger curvature, and

wherein the processor is further configured through execution of the touch event mapping instructions to determine the adjusted touch event position based on the touch event mapping model and the data indicative of the initial assessment of touch event position.