Method and device for reducing latency for rendering and displaying stylus strokes

• US 9,529,463 B1
• Filed: 09/24/2014
• Issued: 12/27/2016
• Est. Priority Date: 09/24/2014
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Prado, Gustavo Diaz, Riding, Mark, Garrido, Erin Rowland, Santoso, Agus, Ramani, Ishwar VenkataManikanda, Albanowski, Kenneth James, Moscovich, Tomer, Pillai, Biju Balakrishna
• Primary Examiner(s): Nguyen, Kevin M
• Application Number:
  US14/495,350
• Time in Patent Office: 825 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G09G 2320/0252 Improving the response speed
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players

Abstract

Devices, systems, and, methods are disclosed for processing stylus interactions with a device and drawing the results of those interactions in a manner that reduces lag. This includes the introduction of a separate overlay module layer that can be updated separately from a normal view system/process of a computing device. In this respect, the overlay module layer may be used to remove unnecessary synchronization events to allow for quick display of stylus input events in the overlay module layer while still allowing the normal rendering process of the operating system to be followed.

Claims

1. A computer-implemented method for displaying strokes, the method comprising:

detecting contact between a stylus and a surface of a display of a computing device, wherein the detecting occurs between a first display frame refresh and a second display frame refresh of the display consecutively following the first display frame refresh;

determining first input data corresponding to a location and time of the contact;

determining first stroke data corresponding to the first input data, the first stroke data being used to display a first stroke corresponding to the contact;

storing the first stroke data in a first buffer;

displaying the first stroke as a temporary stroke using a first hardware pipeline, wherein the first hardware pipeline includes one or more components for performing graphical post processing functions on the first stroke data, the temporary stroke being based on the first stroke data, where the temporary stroke is displayed during the second display frame refresh;

determining, by an application operating on the computing device and substantially in parallel with the determining the first stroke data, second stroke data corresponding to the first input data, the second stroke data being used to display a second stroke corresponding to the contact;

storing the second stroke data in a second buffer;

displaying the second stroke as a processed stroke using a second hardware pipeline separate from the first hardware pipeline, wherein the second hardware pipeline includes one or more components for performing graphical post processing functions on the second stroke data, wherein the processed stroke is displayed during a third display frame refresh, and the processed stroke is displayed at substantially a same location on the display as the temporary stroke; and

removing the temporary stroke from the display during the third display frame refresh.

2. The computer-implemented method of claim 1, wherein:

detecting contact between the stylus and the surface of the display includes;

detecting a first contact between the stylus and the surface of the display, and

detecting a second contact between the stylus and the surface of the display; and

the first input data corresponds to a location and time of the first and second contacts.

3. A computer-implemented method, comprising:

detecting, by a computing device, contact between a stylus and a surface of a display of a computing device between a first display frame refresh and a second display frame refresh of the display consecutively following the first display frame refresh;

determining, by the computing device, first data corresponding to a location and time of the contact;

determining, by the computing device, first stroke data corresponding to the first data, the first stroke data being used to display a first stroke corresponding to the contact;

displaying, by the computing device, the first stroke using a first hardware pipeline during the second display frame refresh; and

processing, by an application of the computing device and substantially in parallel with the determining the first stroke data, the first data to determine a second stroke corresponding to the first data, the second stroke to replace the first stroke and be displayed on the display after the first stroke is displayed.

4. The computer-implemented method of claim 3, further comprising:

storing the first stroke data in a first buffer;

and

processing the first stroke data in the first buffer by the first hardware pipeline between the first display frame refresh and the second display frame refresh.

5. The computer-implemented method of claim 3, further comprising displaying the second stroke during a third display frame refresh, the third display frame refresh occurring after the second display frame refresh.

6. The computer-implemented method of claim 3, further comprising displaying the second stroke using a second hardware pipeline separate from the first hardware pipeline.

7. The computer-implemented method of claim 3, further comprising, during a same frame refresh, displaying the second stroke on the display and removing the first stroke from the display.

8. The computer-implemented method of claim 7, wherein the second stroke is displayed at substantially a same location on the display as the first stroke.

9. The computer-implemented method of claim 3, further comprising:

detecting, by the computing device, second contact between a stylus and a surface of a display of the computing device;

determining, by the computing device, second data corresponding to the second contact; and

delaying determining, by the computing device, third stroke data corresponding to the second data to account for latency in determining the second data, the third stroke data being used to display a third stroke corresponding to the second contact.

10. The computer-implemented method of claim 3, further comprising sending, by an overlay component of the computing device, the first data to the application for processing.

11. A computing device, comprising:

a display;

at least one processor;

a memory device including instructions operable to be executed by the at least one processor to perform a set of actions, configuring the at least one processor to;

detect contact between a stylus and a surface of a display of a computing device between a first display frame refresh and a second display frame refresh of the display consecutively following the first display frame refresh;

determine first data corresponding to a location and time of the contact;

determine first stroke data corresponding to the first data, the first stroke data being used to display a first stroke corresponding to the contact;

display the first stroke using a first hardware pipeline during the second display frame refresh; and

process, substantially in parallel with the determining the first stroke data, the first data to determine a second stroke corresponding to the first data, the second stroke to replace the first stroke and be displayed on the display after the first stroke is displayed.

12. The computing device of claim 11, wherein the at least one processor is further configured to display the second stroke during a third display frame refresh, the third display frame refresh occurring after the second display frame refresh.

13. The computing device of claim 11, wherein the at least one processor is further configured to:

store the first stroke data in a first buffer; and

process the first stroke data in the first buffer by the first hardware pipeline between the first display frame refresh and the second display frame refresh.

14. The computing device of claim 11, wherein the at least one processor is configured to display the second stroke using a second hardware pipeline separate from the first hardware pipeline.

15. The computing device of claim 11, wherein the at least one processor is further configured to, during a same frame refresh, display the second stroke on the display and remove the first stroke from the display.

16. The computing device of claim 15, wherein the second stroke is displayed at substantially a same location on the display as the temporary stroke.

17. The computing device of claim 11, the at least one processor is configured to allocate processing resources to allow the contact to be detected and the first stroke displayed within a refresh rate of the computing device.

18. The computing device of claim 11, wherein the at least one processor includes an overlay component and an application component and the overlay component is configured to send the first data to the application component to process the first data to determine the second stroke.