Recognizing gestures and updating display by coordinator

• US 10,761,714 B2
• Filed: 11/23/2015
• Issued: 09/01/2020
• Est. Priority Date: 11/23/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Fischer, Ian, Garde, Chinmay Sunil
• Primary Examiner(s): Chavez, Renee D
• Assistant Examiner(s): Stanley, Jeremy L
• Application Number:
  US14/949,387
• Publication Number:
  US 20170147176A1
• Time in Patent Office: 1,744 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A non-transitory computer-readable storage medium may comprise instructions stored thereon. When executed by at least one processor, the instructions may be configured to cause a computing device to implement at least a user interface module and a coordinator module. The user interface module may be configured to pass layout constraints to the coordinator module. The coordinator module may be configured to process input received by the computing device, recognize gestures based on the input, update a display based on the recognized gestures and the layout constraints passed to the coordinator module by the user interface module, and in response to recognizing an application-specific gesture, pass the application-specific gesture to the user interface module.

Claims

1. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing device to implement at least:

a user interface thread configured to;

pass layout constraints to a coordinator thread;

pass conditions to the coordinator thread defining when the coordinator thread recognizes an application-specific gesture and wakes up the user interface thread, wherein at least one of the conditions is a change to a default condition, wherein the conditions defining when the coordinator thread recognizes the application-specific gesture include conditions defining gestures which prompt an action by an application;

after passing the layout constraints and the conditions to the coordinator thread, go to sleep; and

in response to receiving the application-specific gesture from the coordinator thread and being prompted to wake up by the coordinator thread, wake up and prompt the application to perform the action based on the application-specific gesture; and

the coordinator thread configured to;

process a first input received by the computing device;

determine that the first input is not the application-specific gesture based on a location on a display of the computing device of the first input and a user interface element receiving the first input and based on the conditions defining when the coordinator thread recognizes the application-specific gesture;

update the display, while the user interface thread is sleeping, based on the first input, the determination that the first input is not the application-specific gesture, and the layout constraints passed to the coordinator thread by the user interface thread;

wherein updating the display includes changing an appearance of the user interface element on the display without prompting the application;

process a second input received by the computing device;

determine that the second input is the application-specific gesture based on the conditions defining when the coordinator thread recognizes the application-specific gesture passed by the user interface thread; and

in response to determining that the second input is the application-specific gesture, pass the application-specific gesture to the user interface thread, the passing the application-specific gesture to the user interface thread prompting the user interface thread to wake up.

2. The non-transitory computer-readable storage medium of claim 1, wherein the layout constraints define at least a location of a first user interface element on the display with respect to a location of a second user interface element on the display.

3. The non-transitory computer-readable storage medium of claim 1, wherein the layout constraints define constraints between user interface elements on the display without describing a control flow.

4. The non-transitory computer-readable storage medium of claim 1, wherein the user interface thread is further configured to construct the layout constraints based on at least one hardware specification of the display.

5. The non-transitory computer-readable storage medium of claim 1, wherein the user interface thread is further configured to construct the layout constraints based on a pixel width of the display and a pixel length of the display.

6. The non-transitory computer-readable storage medium of claim 1, wherein the input is received via a touchscreen included in the computing device.

7. The non-transitory computer-readable storage medium of claim 1, wherein the updating the display includes determining a location on the display of at least one user interface element.

8. The non-transitory computer-readable storage medium of claim 1, wherein the updating the display includes solving layout and gesture constraints.

9. The non-transitory computer-readable storage medium of claim 1, wherein the updating the display includes solving layout and gesture constraints based on a Cassowary linear arithmetic constraint solving algorithm.

10. The non-transitory computer-readable storage medium of claim 1, wherein the application-specific gesture includes a gesture that prompts a response from the application.

11. The non-transitory computer-readable storage medium of claim 1, wherein the application-specific gesture includes a click on a button associated with the application.

12. The non-transitory computer-readable storage medium of claim 1, wherein the user interface thread is further configured to prompt the application to perform an action in response to receiving the application-specific gesture from the coordinator thread.

13. The non-transitory computer-readable storage medium of claim 1, wherein the user interface thread is further configured to pass, to the coordinator thread, conditions for the coordinator thread to wake up the user interface thread.

14. The non-transitory computer-readable storage medium of claim 1, wherein the user interface thread is further configured to inform the coordinator thread of constraints between properties of future input events and user interface elements.

15. A computing device comprising:

a touchscreen configured to display user interface elements and receive touch input;

at least one processor; and

at least one memory device comprising instructions stored thereon that, when executed by the at least one processor, are configured to cause the computing device to implement;

a user interface thread configured to;

pass layout constraints to a coordinator thread;

pass conditions to the coordinator thread defining when the coordinator thread recognizes an application-specific gesture and wakes up the user interface thread, wherein at least one of the conditions is a change to a default condition, wherein the conditions defining when the coordinator thread recognizes the application-specific gesture include conditions defining gestures which prompt an action by an application;

after passing the layout constraints to the coordinator thread, go to sleep; and

in response to receiving the application-specific gesture from the coordinator thread and being prompted to wake up by the coordinator thread, wake up and prompt the application to perform the action based on the application-specific gesture; and

the coordinator thread configured to;

process a first touch input received by the touchscreen;

determine that the first touch input is not the application-specific gesture based on a location on the touchscreen of the computing device of the first touch input and a user interface element receiving the first input and based on the conditions defining when the coordinator thread recognizes the application-specific gesture;

update the touchscreen, while the user interface thread is sleeping, based on the first touch input, the determination that the first touch input is not the application-specific gesture, and the layout constraints passed to the coordinator thread by the user interface thread;

wherein updating the touchscreen includes changing an appearance of the user interface element on the touchscreen without prompting the application;

process a second touch input received by the computing device;

determine that the second touch input is the application-specific gesture based on the conditions defining when the coordinator thread recognizes the application-specific gesture passed by the user interface thread; and

in response to determining that the second input is the application-specific gesture, pass the application-specific gesture to the user interface thread, the passing the application-specific gesture to the user interface thread prompting the user interface thread to wake up.

16. The computing device of claim 15, wherein the layout constraints define at least a location of a first user interface element on the touchscreen with respect to a location of a second user interface element on the touchscreen.

17. The computing device of claim 15, wherein the user interface thread is further configured to construct the layout constraints based on a pixel width and a pixel length of the touchscreen.

18. The computing device of claim 15, wherein the application-specific gesture includes a gesture that prompts a response from an application running on the computing device.

19. The computing device of claim 15, wherein the user interface thread is further configured to prompt the application to perform the action in response to receiving the application-specific gesture from the coordinator thread.

20. A method performed by a computing device, the method comprising:

passing, by a user interface thread, layout constraints to a coordinator thread;

passing, by the user interface thread to the coordinator thread, conditions defining when the coordinator thread recognizes an application-specific gesture and wakes up the user interface thread, wherein at least one of the conditions is a change to a default condition, wherein the conditions defining when the coordinator thread recognizes the application-specific gesture include conditions defining gestures which prompt an action by an application;

after passing the layout constraints and the conditions to the coordinator thread, the user interface thread going to sleep;

in response to receiving the application-specific gesture and being prompted to wake up by the coordinator thread, the user interface thread waking up and prompting the application to perform the action based on the application-specific gesture;

processing, by the coordinator thread, a first input received by the computing device;

determining, by the coordinator thread, that the first input is not the application-specific gesture based on a location on a display of the computing device of the first input and a user interface element receiving the first input and based on the conditions defining when the coordinator thread recognizes the application-specific gesture;

updating the display by the coordinator thread, while the user interface thread is sleeping, based on the first input, the determination that the first input is not the application-specific gesture, and the layout constraints passed to the coordinator thread by the user interface thread;

wherein updating the display includes changing an appearance of the user interface element on the display without prompting the application;

processing, by the coordinator thread, a second input received by the computing device;

determining, by the coordinator thread, that the second input is the application-specific gesture based on the conditions defining when the coordinator thread recognizes the application-specific gesture passed by the user interface thread; and

in response to determining that the second input is the application-specific gesture, passing, by the coordinator thread, the application-specific gesture to the user interface thread, the passing the application-specific gesture to the user interface thread prompting the user interface thread to wake up.