Instantiable gesture objects

• US 9,575,652 B2
• Filed: 03/31/2012
• Issued: 02/21/2017
• Est. Priority Date: 03/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Rossi, Jacob S., Rogers, Justin E., Ramanathan, Raju, Wright, Stephen H., Chandiram, Charu, Rakow, Matthew A.
• Primary Examiner(s): Bhatia, Ajay
• Assistant Examiner(s): Chung, Mong-Shune
• Application Number:
  US13/436,877
• Publication Number:
  US 20130263029A1
• Time in Patent Office: 1,788 Days
• Field of Search: 715/764
• US Class Current: 1/1
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: H04N 1/00411 the display also being used for user input, e.g. touch screen

Abstract

Instantiable gesture object techniques are described in which native gesture functionality is abstracted to applications using a script-based recognition interface. Gesture objects may be instantiated for different interaction contexts at the direction of applications programmed using dynamic scripting languages. Gesture objects can be configured to designate particular touch contacts and/or other inputs to consider for gesture recognition and a target element of content to which corresponding recognized gestures are applicable. After creation, gesture objects manage gesture processing operations on behalf of the applications including creating recognizers with the native gesture system, feeding input data for processing, and transforming raw gesture data into formats appropriate for the application and/or a target element. Accordingly, script-based applications may use the gesture objects to offload processing tasks associated with gesture recognition and take advantage of native gesture functionality.

Claims

1. A method comprising:

instantiating a gesture object for an application to handle gesture recognition for the application through native gesture functionality provided by a computing device;

associating the gesture object with interaction inputs and a target element specified by the application such that the interaction inputs directed to the target element are offloaded to the gesture object configured for the application, the target element representing a selectable element rendered by the computing device;

creating a recognizer on behalf of the application to facilitate gesture recognition through the native gesture functionality provided by the computing device;

feeding interaction input data for the interaction inputs to the recognizer to enable recognition of gestures corresponding to the application based on the interaction input data;

obtaining gesture event messages from the recognizer that are indicative of recognized gestures for the application;

processing raw gesture data described by the gesture event messages on behalf of the application using the gesture object; and

firing gesture events having processed gesture data to the associated target element in accordance with a content model for the application such that the recognized gestures conveyed via the gesture event messages are applied to the target element.

2. A method as described in claim 1, wherein the gesture object is a script-based abstraction of underlying native gesture functionality provided by the computing device.

3. A method as described in claim 1, wherein the application is programmed using a dynamic scripting language.

4. A method as described in claim 1, wherein the method is performed via an interface exposed by a rendering engine employed by the application to render content for the application.

5. A method as described in claim 1, wherein the interaction inputs comprise touch contacts applied to a touchscreen associated with the computing device.

6. A method as described in claim 1, wherein the processing comprises transforming the raw gesture data to a coordinate space specified by the application.

7. A method as described in claim 1, wherein the recognizer is configured to track interaction inputs designated by the gesture object and map the interaction inputs to a library of supported gestures to recognize when particular gestures occur.

8. A method as described in claim 1, wherein the gesture object is configured to translate data for recognized gestures between a system format used by the recognizer and a script-based format understandable by the application.

9. A method as described in claim 1, wherein the interaction inputs are configured as pinning references to the gesture object that prevent the gesture object from collecting itself as long as the interaction inputs remain active.

10. A method as described in claim 1, wherein the gesture events fired to the target elements contain information that identifies the gesture object to enable continued access the gesture object when reference to the gesture object is lost.

11. A method as described in claim 1, wherein associating the target element with the gesture object causes gesture events generated in relation to the gesture object to be explicitly directed to the target element thereby enabling the application to directly apply the gesture events to the target element.

12. A computing device comprising:

a processing system;

a gesture module operable via the processing system to provide native gesture recognition functionality for the computing device, the native gesture recognition functionality representing gesture recognition functionality made accessible to applications via an operating system of the computing device; and

a recognition interface operable via the processing system to handle gesture recognition for applications that use dynamic scripting language;

including;

registering the applications for gesture recognition with the gesture module via gesture objects that are instantiated to represent multiple interaction contexts, each gesture object configured to specify one or more interaction inputs and a target element of application content for a corresponding one of the multiple interaction contexts;

supplying input data for the interaction inputs to the gesture module for processing on behalf of the applications;

transforming raw gesture data obtained from the gesture module that provides the native functionality for recognized gestures into coordinate systems employed by target elements specified by the gesture objects; and

communicating the transformed gesture data for use by the applications to manipulate display of the target elements in accordance with the recognized gestures, the transformed gesture data being usable by the applications in the dynamic scripting language.

13. A computing device as described in claim 12, further comprising a rendering engine configured to render content for the applications, wherein the recognition interface is exposed to the applications via the rendering engine.

14. A computing device as described in claim 13, wherein the rendering engine is implemented as a component of a browser.

15. A computing device as described in claim 13, wherein the rendering engine is configured to parse the content for the applications to represent the content in accordance with a document object model (DOM), the target elements specifying a mapping of the gesture objects to particular elements in one or more DOMs representing the content.

16. A computing device as described in claim 12, wherein the interaction inputs specified for one or more of the gesture objects comprise inputs from a combination of multiple input sources.

17. A computing device as described in claim 12, wherein the recognition interface implements the gesture objects as a built-in object type that is exposed for use by applications that use dynamic scripting language.

18. One or more computer-readable storage media storing instructions that, when executed via a computing device, cause the computing device to implement an dynamically compiled application configured to perform operations including:

directing creation of a gesture object for a selected target element of content for the application to register for gesture detection via a script-based recognition interface exposed by a rendering engine of the computing device;

obtaining gesture events fired on the target element by the gesture object that are indicative of gestures recognized based upon user interaction with the target element;

tracking a state of the target element;

applying the gesture events to the target element to manipulate display of the target element in accordance with the recognized gestures; and

updating the state of the target element based on the applying.

19. One or more computer-readable storage media as described in claim 18, wherein the directing creation of the gesture object comprises setting properties of the gesture object to specify the selected target element and one or more interaction inputs to be considered for recognition of gestures to apply to the selected target element.

20. One or more computer-readable storage media as described in claim 18, wherein the gesture object is configured to handle feeding of input data to a native gesture system of the computing device and processing of gesture data supplied back by the native gesture system on behalf of the application such that the processed gesture data is in a script format understandable by the application.