Optimization schemes for controlling user interfaces through gesture or touch

• US 10,762,277 B2
• Filed: 09/24/2014
• Issued: 09/01/2020
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Devi, Renu, Whitfield, Barnabas, Ryan, Kyle, Harris, Trevor, Fields, Mark, Herzfeld, Michael, Lee, Louisa
• Primary Examiner(s): Ipakchi, Maryam M
• Application Number:
  US14/494,591
• Publication Number:
  US 20150012815A1
• Time in Patent Office: 2,169 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/14 Tree-structured documents parsing G06F40/205; validation G06F40/226
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A web application provides a custom selection for editing text on a gesture or touch screen. The application replaces native browser handles with selection handles to provide consistent user interface experience across platforms. The application also provides a scheme for semantic interpretation of browser gesture or touch events. The application standardizes browser events into a consistent stream of semantic events that are compatible with a plurality of devices and browsers. The application also provides a gesture or touch optimized user interface in the browser. The application determines gesture or touch input and optimizes the user interface according to the type of input.

Claims

1. A method executed on a computing device for transforming browser events into a semantic event comprehensible by an application executed by a remote server and hosted by a browser, the method comprising:

receiving, from the browser, a browser event generated in response to a user input received via a user interface outputted on the browser that is executed on the computing device, the browser event representing an input configured to initiate a browser action to handle a selection with an operating system of the computing device, the browser action being browser or computing device dependent;

upon receiving the browser event, determining, at the computing device, whether the received browser event matches a predetermined input event that corresponds to a semantic event of the application, the semantic event representing a command comprehensible by the application hosted by the browser and executed by the remote server;

in response to determining that the received browser event matches the predetermined input event,

transmitting, from the computing device and via a computer network, the semantic event comprehensible by the application to the application executed by the remote server;

cancelling, in the browser, the received browser event on the computing device to prevent the browser action that is browser or computing device dependent from being performed on the computing device;

receiving, via the computer network, an object generated by the application executed on the remote server in response to the command represented by the semantic event; and

rendering, at the computing device and via the interface in the browser, the received object generated by the application in response to the user input in the browser on the computing device.

2. The method of claim 1, wherein receiving the browser event generated in response to input received via the touch enabled device and determining whether the received browser event matches the predetermined input event comprises:

receiving, at a register event handler of an event handler manager, the browser event;

providing, through the register event handler, the browser event to at least one input manager associated with the browser event;

processing, by the at least one associated input manager, the browser event and a document object model element of the browser event; and

identifying, based on the processing, the semantic event associated with the browser event,

wherein providing, to the application, the semantic event comprehensible by the application comprises providing the semantic event through the event handler manager.

3. The method of claim 1, wherein:

the browser event comprises a series of browser events and the predetermined input event comprises a predetermined series of browser events,

cancelling, in the browser, the browser event comprises cancelling, in the browser, the series of browser events to prevent corresponding browser actions in the browser.

4. The method of claim 3, wherein the semantic event comprises a stream of semantic events emulating the series of browser events.

5. The method of claim 4, further comprising:

ordering the series of browser events based at least in part on corresponding priority values.

6. The method of claim 5, further comprising:

for the browser events with an equal priority, providing the browser events of the series in a random order.

7. The method of claim 4, further comprising:

notifying the application about the stream of semantic events at an input manager associated with the stream; and

notifying other input managers to stop listening for associated semantic events.

8. A computing system for transforming a browser event into a semantic event comprehensible by an application hosted by a browser and executed by a remote server, the computing system comprising:

a memory storing computer-executable instructions of;

receiving, from the browser, a browser event generated in response to a user input received via an interface of an application outputted by the browser, the browser event representing an input configured to initiate a browser action to handle a selection with an operating system of the computing device, wherein the browser action is browser or computing system dependent;

upon receiving the browser event, determining, whether the received browser event matches a predetermined input event corresponds to a semantic event of the application, the semantic event representing a command comprehensible by the application executed by the remote server;

providing, to the application executed on the remote server, the semantic event from the computing system and via a computer network;

cancelling, in the browser, the browser event to prevent the browser action from being performed on the computing system;

receiving, via the computer network, an object generated by the application executed on the remote server in response to the command represented by the semantic event; and

rendering, via the interface of the application in the browser, the received object generated by the application executed on the remote server in response to the semantic event; and

a processor for executing the computer-executable instructions stored in the memory.

9. The computing system of claim 8, wherein the browser event comprises a series of browser events and the predetermined input event comprises a predetermined series of browser events,

wherein cancelling, in the browser, the browser event comprises cancelling, in the browser, the series of browser events to prevent corresponding browser actions from being performed on the computing system.

10. The computing system of claim 9, wherein the semantic event comprises a stream of semantic events emulating the series of browser events.

11. The computing system of claim 8, wherein the semantic event associated with the browser event is determined based at least in part on a type of input associated with the browser event.

12. The computing system of claim 11, wherein the type of input comprises a gesture or touch.

13. The computing system of claim 11, wherein the type of input comprises a stylus input.

14. The computing system of claim 11, wherein the type of input comprises a mouse click.

15. A method executed on a computing device for transforming browser events into a semantic event comprehensible by a web application executed by a remote server and hosted by a browser on the computing device, the method comprising:

upon receiving, in response to a user input to a user interface of the application on the browser, a browser event representing an input configured to initiate a browser action that is browser dependent or computing device dependent,

determining, at the computing device, whether the received browser event matches a predetermined input event corresponds to a semantic event of the application, the semantic event representing a command comprehensible by the application hosted by the browser and executed by the remote server; and

in response to determining that the received browser event matches the predetermined input event,

transmitting, from the computing device and via a computer network, the command represented by the semantic event to the remote server in lieu of performing the browser action on the computing device;

receiving, via the computer network, an object generated by the application executed on the remote server in response to the command represented by the semantic event; and

rendering, at the computing device and via the interface in the browser, the received object generated by the application in response to the user input in the browser on the computing device.

16. The method of claim 15, further comprising providing an indication of the browser event and a type of input of the browser event to the application, wherein the type of input comprises a gesture, a touch, a stylus input, or a mouse click.

17. The method of claim 15, wherein the browser event comprises a series of browser events and the semantic event comprises a stream of semantic events emulating the series of browser events.

18. The method of claim 15, further comprising cancelling, in the browser, the browser event to prevent the corresponding browser action from being performed on the computing device in response to determining that the received browser event matches the predetermined input event.

19. The method of claim 15, wherein:

the browser event includes multiple browser sub-events; and

determining whether the received browser event matches the predetermined input event includes determining whether the multiple browser sub-events represent a pattern that matches the predetermined input event.

20. The method of claim 15, wherein:

the browser event includes multiple browser sub-events;

determining whether the received browser event matches the predetermined input event includes determining whether the multiple browser sub-events represent a pattern that matches the predetermined input event; and

in response to determining that the multiple browser sub-events represent a pattern that matches the predetermined input event, cancelling, in the browser, the multiple browser sub-events to prevent corresponding browser actions from being performed on the computing device.