Predictive compensation for a latency of an input device

• US 10,452,188 B2
• Filed: 01/13/2012
• Issued: 10/22/2019
• Est. Priority Date: 01/13/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Santiago, Pravin K., Kaneko, Masahiko, Letie, Nikola, Olsen, Thomas W., Li, Ruoyang
• Primary Examiner(s): Sasinowski, Andrew
• Application Number:
  US13/350,588
• Publication Number:
  US 20130181908A1
• Time in Patent Office: 2,839 Days
• Field of Search: 345173
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A computer configured to selectively use prediction to compensate for a latency of an input device. The input device may be a touch screen. Processing of touch inputs may lead to a latency between the user performing a touch input and the touch screen display responding to the touch input. A touch screen component predicts a subsequent position of a touch contact point based on the sensed positions of the touch contact point. The component may be a software component that provides a predictive output that can be selectively used based on a characteristic of a touch input. The component may be generic such that it may be applied on multiple types of computers equipped with different types of touch sensing hardware configured with any suitable settings. The component'"'"'s output may be used to control the touch screen display.

Claims

1. A method of predictively compensating for a latency associated with a touch screen of a computer, the method comprising:

receiving, within the computer, at time t₁, first reporting of a first sensed position of a touch;

predicting, by the computer, a position of the touch at time t₂, based at least in part on characteristics of a touch sensing unit of the computer and the first sensed position of the touch first reported at time t₁, where t₂ is later than t₁;

outputting, internally within the computer, the predicted position of the touch at time t₂ as a potential alternative for programmatic selection over a second sensed position of the touch second reported at time t₂;

determining a type of the touch, the type being one of a tap, a press, or a swipe; and

upon determining that the type of the touch is a first type comprising one of the tap, the press, or the swipe, displaying the predicted position of the touch at time t₂; and

upon determining that the type of the touch is a second type comprising one of the tap, the press, or the swipe, displaying the second sensed position of the touch second reported at time t₂ wherein the first type is not the same as the second type.

2. The method of claim 1, wherein the programmatic selection between the predicted position of the touch at time t₂ and the second sensed position of the touch second reported at time t₂ is performed by at least one of a touch processing component associated with the touch screen or an application.

3. The method of claim 1, wherein the programmatic selection between the predicted position of the touch at time t₂ and the second sensed position of the touch second reported at time t₂ is associated with a pan command.

4. The method of claim 1, wherein predicting a position of the touch at time t₂ comprises:

calculating a plurality of prediction parameters; and

calculating, based at least in part on the plurality of prediction parameters and the characteristics of the touch sensing unit of the computer, the predicted position of the touch at time t₂.

5. The method of claim 4, wherein calculating the plurality of prediction parameters comprises calculating at least one of a plurality of trajectory parameters associated with the first sensed position first reported at time t₁, a smoothing parameter, a limit on the predicted position of the touch at time t₂, and a plurality of parameters associated with an adaptive filter.

6. The method of claim 4, wherein calculating the predicted position of the touch at time t₂ comprises:

calculating a plurality of predicted changes in position of the touch;

smoothing the plurality of predicted changes in position of the touch; and

calculating a candidate predicted position of the touch by combining the plurality of predicted changes in position of the touch with a base position of the touch.

7. The method of claim 6, wherein the candidate predicted position of the touch is the predicted position of the touch at time t₂.

8. The method of claim 6, wherein:

calculating the predicted position of the touch at time t₂ further comprises selecting, as the predicted position of the touch at time t₂, one of the candidate predicted position of the touch and a prior predicted position of the touch,

the prior predicted position of the touch is associated with a time prior to the time associated with the candidate predicted position, and

the selecting of the predicted position of the touch at time t₂ is based on a direction of movement associated with the first sensed position first reported at time t₁ and on a direction of movement between the prior predicted position and the candidate predicted position.

9. The method of claim 1, wherein the first type is a different type than the second type.

10. An apparatus for predictively compensating for a latency associated with a touch screen of a computer, the computer including a touch sensing unit, a display, and a controller of the display, the touch sensing unit configured to sense a touch of the touch screen and to provide a plurality of sensed positions of the touch, the display controller configured to control the display based on a selected plurality of positions, the apparatus comprising:

a processing unit; and

a memory storing instructions which, when executed by the processing unit, cause the apparatus to perform a method of;

receiving at time t₁, by a touch processing component, from the touch sensing unit, first reporting of a first sensed position of a touch;

predicting, by the touch processing component, a position of the touch at time t₂, based at least in part on characteristics of the touch sensing unit of the computer and the first sensed position of the touch first reported at time t₁, where t₂ is later than t₁, said first sensed position first reported at time t₁ is filtered by a first filter and said predicted position of the touch at time t₂ extending beyond the first sensed position of the touch first reported at time t₁;

programmatically making a selection, by the touch processing component, between the predicted position of the touch at time t₂ and a second sensed position of the touch second reported at time t₂, based at least in part, on a type of the touch and on comparing a confidence measure associated with an accuracy of the predicted position of the touch at time t₂ with a confidence threshold, wherein the type of the touch is one of a tap, a press, or a swipe, said predicted position of the touch at time t₂ being further limited according to a second filter; and

controlling the display based on the programmatic selected one of the predicted position of the touch at time t₂ when the type of the touch is a first type comprising one of the tap, the press, or the swipe, and the second sensed position of the touch second reported at time t₂ when the type of the touch is a second type comprising one of the tap, the press, or the swipe, wherein the first type is not the same as the second type.

11. The apparatus of claim 10, further comprising providing, by the touch processing component, the predicted position of the touch at time t₂, the second sensed position second reported at time t₂, and an indication of the selection, to a software component for the software component to make the programmatic selection.

12. The apparatus of claim 11, further comprising the software component programmatically selecting the predicted position of the touch at time t₂ regardless of whether the touch processing component selected the predicted position of the touch at time t₂.

13. The apparatus of claim 10, wherein predicting the position of the touch at time t₂ comprises:

calculating a plurality of prediction parameters based on at least in part on a characteristic of the touch sensing unit and a characteristic of the touch; and

calculating the position of the touch at time t₂, based at least in part on the plurality of prediction parameters.

14. The apparatus of claim 13, wherein calculating the plurality of prediction parameters comprises calculating at least one of a velocity of the touch, an acceleration of the touch, a maximum acceleration of the touch, or a smoothing parameter, based on the first sensed position first reported at time t₁, and on a latency associated with the touch sensing unit.

15. The apparatus of claim 13, wherein calculating the plurality of prediction parameters comprises calculating a limit on the predicted position of the touch at time t₂ based on the first sensed position of the touch at time t₁, a latency associated with the touch sensing unit, and a sample time associated with the touch sensing unit.

16. The apparatus of claim 13, wherein calculating the plurality of prediction parameters comprises calculating a plurality of parameters associated with an adaptive filter based on the first sensed position of the touch at time t₁ and the predicted position of the touch at time t₂.

17. A method of predictively compensating for a latency associated with an input device of a computer, the computer including an input sensing unit, an output device, and a controller of the output device, the input sensing unit configured to sense input and to provide a plurality of sensed inputs including a first sensed input first provided at time t₁, the output device controller configured to control the output device based on a selected plurality of inputs, the method comprising:

obtaining a plurality of characteristic parameters of the input sensing unit;

calculating a plurality of prediction parameters of an input processing component based on the plurality of characteristic parameters of the input sensing unit and on the first sensed input first provided at time t₁, said first sensed input first provided at time t₁ being filtered by a first filter;

obtaining additional data associated with the first sensed input first provided at time t₁, wherein the additional data includes at least one of a number of contact points of the first sensed input, a duration of the first sensed input, and a command associated with the first sensed input;

adapting the plurality of prediction parameters, said predicted parameters being further limited according to a second filter, based on the additional data associated with the first sensed input;

predicting an input at time t₂, where t₂ is later than t₁, based on the plurality of adapted prediction parameters, which were initially calculated based on the plurality of characteristic parameters of the input sensing unit, and on the first sensed input provided at time t₁; and

programmatically selecting between the predicted input at time t₂ and a second sensed input second provided at time t₂, based, at least in part, on the following;

comparing a confidence measure associated with an accuracy of the plurality of prediction parameters with a confidence threshold; and

determining a type of the first sensed input, the type being one of a tap, a press, or a swipe; and

upon determining that the type of the first sensed input is a first type comprising one of the tap, the press, or the swipe, displaying the predicted input at time t₂; and

upon determining that the type of the first sensed input is a second type comprising one of the tap, the press, or the swipe, displaying the second sensed input provided at time t₂ wherein the first type is not the same as the second type.

18. The method of claim 17, wherein obtaining a plurality of characteristic parameters of the input sensing unit comprises:

receiving a latency associated with the input sensing unit; and

sensing a sample time associated with the input sensing unit.

19. The method of claim 17, wherein the predicted input to be provided at time t₂ accounts for latency associated with the input device.

20. The method of claim 17, wherein the first type is a different type than the second type.