Multi-threaded conditional processing of user interactions for gesture processing using rendering thread or gesture processing thread based on threshold latency

• US 9,069,459 B2
• Filed: 05/03/2011
• Issued: 06/30/2015
• Est. Priority Date: 05/03/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Driver, David Owen, Lobo, Philomena Francisca, Liu, Xun
• Primary Examiner(s): An, Meng
• Assistant Examiner(s): GHAFFARI, ABU Z
• Application Number:
  US13/099,388
• Publication Number:
  US 20120284663A1
• Time in Patent Office: 1,519 Days
• Field of Search: 718/102, 715/784, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/501 Performance criteria
• CPC Class Codes: G06F 2209/5018 Thread allocation
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/505 considering the load

Abstract

Embodiments isolate gesture processing for improved user interactivity with content. A first thread detects user interaction with a document via a touch sensor associated with a computing device. A latency associated with a second thread is calculated. The calculated latency is compared to a threshold latency. Based on the comparison, the first thread determines whether to transfer processing of the detected user interaction from the first thread to the second thread. The detected user interaction is processed by the first thread or by the second thread based on the determination. In some embodiments, users are provided with smooth interactivity such as panning and scrolling when browsing web content with a browser executing on a mobile computing device.

Claims

1. A system for separating gesture processing from rendering processing, said system comprising:

a memory area associated with a mobile computing device, said memory area storing at least a portion of a web page and one or more of a plurality of scrollable elements associated therewith, said memory area further storing a threshold latency, the threshold latency being based upon a noticeable delay in interaction with the mobile computing device;

a processor programmed to;

detect, by a gesture processing thread via a touch sensor associated with the mobile computing device, a request to scroll the web page stored in the memory area;

request, from a rendering thread, touch point data associated with the detected request, said touch point data identifying one of the scrollable elements associated with the detected request;

calculate an elapsed time since requesting the touch point data and receipt of the touch point data;

scroll, by the rendering thread, said one of the scrollable elements when the calculated elapsed time is within the threshold latency stored in the memory area; and

scroll, by the gesture processing thread, the web page when the calculated elapsed time is not within the threshold latency stored in the memory area, wherein the rendering thread corresponds to a slow response thread and the gesture processing thread corresponds to a fast response thread.

2. The system of claim 1, wherein the processor is further programmed to request the touch point data by requesting at least one of whether a touch point is scrollable, whether the touch point is editable, or a shape of the touch point.

3. The system of claim 1, wherein the plurality of scrollable elements comprises at least one of a frame, a subpane, a panel, or a popup.

4. The system of claim 1, further comprising means for separating gesture processing from rendering processing.

5. The system of claim 1, wherein scrolling by the gesture processing thread includes analyzing the touch point data to correlate the touch point data with a gesture and to determine an operation to be performed on the portion of a web page stored in the memory area based on the gesture.

6. The system of claim 1, wherein scrolling by the rendering thread comprises performing an operation on a current version of the web page.

7. A method comprising:

detecting, by a first thread, user interaction with a document via a touch sensor associated with a computing device, said first thread handling gesture processing;

calculating a latency associated with a second thread, said second thread handling rendering of the document, wherein calculating the latency comprises requesting, from the second thread, touch point data associated with the detected user interaction, calculating an elapsed time since said requesting, and determining the latency based at least on the calculated elapsed time;

comparing the calculated latency to a threshold latency, the threshold latency being based on a perceivable delay in the user interaction;

determining, based on the comparison, whether to transfer processing of the detected user interaction from the first thread to the second thread;

processing the detected user interaction by the second thread by scrolling a portion of the document when the calculated latency is less than the threshold latency and the second thread provides the requested touch point data; and

processing the detected user interaction by the first thread by scrolling the entire document when the calculated latency is greater than the threshold latency and the second thread fails to provide the requested touch point data.

8. The method of claim 7, further comprising:

receiving, by the first thread, the requested touch point data from the second thread; and

determining an operation associated with the received touch point data.

9. The method of claim 8, wherein determining the operation comprises identifying a gesture associated with the received touch point data.

10. The method of claim 7, wherein processing the detected user interaction comprises processing, by the first thread, the detected user interaction if the calculated latency is greater than the threshold latency.

11. The method of claim 7, wherein processing the detected user interaction comprises performing, by the second thread, the operation if the calculated latency is less than the threshold latency.

12. The method of claim 7, wherein the detected user interaction corresponds to at least one operation, and wherein processing the detected user interaction by the first thread comprises applying, by the first thread, the operation to a version of the document cached by the computing device.

13. The method of claim 7, wherein the detected user interaction corresponds to at least one operation, and wherein processing the detected user interaction by the second thread comprises applying, by the second thread, the operation to a version of the document stored by another computing device associated with the computing device.

14. The method of claim 7, wherein detecting the user interaction comprises detecting user interaction with a web page.

15. The method of claim 7, wherein requesting touch point data comprises requesting at least one of whether a touch point is scrollable, whether the touch point is editable, or a shape of the touch point.

16. The method of claim 7, wherein the touch point data identifies at least one of a frame, or a scrollable element.

17. One or more computer readable storage media having computer-executable components, said components comprising:

a user interface component that when executed causes at least one processor to detect, by a fast response thread, user interaction with a document via a touch sensor associated with a computing device, said fast response thread handling gesture processing;

a touch point component that when executed causes at least one processor to request, from a slow response thread by the fast response thread, touch point data corresponding to the user interaction detected by the user interface component, said slow response thread handling display of the document, said requested touch point data being associated with an operation to be applied to at least a portion of the document;

a measurement component that when executed causes at least one processor to compare, by the fast response thread, a latency associated with the slow response thread to a threshold latency, the threshold latency being based on a perceivable delay in the user interaction; and

an overlay component that when executed causes at least one processor to apply, based on the comparison performed by the measurement component, the operation by the fast response thread to the entire document or by the slow response thread to only said portion of the document, wherein the overlay component applies the operation by the fast response thread to the entire document when the latency violates the threshold latency and applies the operation by the slow response thread to only said portion of the document when the latency satisfies the threshold latency.

18. The computer readable storage media of claim 17, wherein the user interaction corresponds to a gesture and the operation includes a scrolling operation.

19. The one or more computer readable storage media of claim 17, wherein the touch point data identifies one or more scrollable elements associated with the document.

20. The one or more computer readable storage media of claim 17, wherein the slow response thread applies the operation to a live portion of the document.