CROSS-PLATFORM INTERACTIVITY ARCHITECTURE

• US 20170277381A1
• Filed: 07/29/2016
• Published: 09/28/2017
• Est. Priority Date: 03/25/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Allyn, Barry Christopher
• Granted Patent: US 11,029,836 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

Maintaining an application for deployment across platforms requires significant engineering effort and an expanded code base, especially when dealing with different input types from the host platforms. To reduce the computational expenses and improve the standardization of the user interface, systems and methods are provided to recognize gestures in a platform agnostic code architecture, which may be deployed on multiple platforms with different input and output methodologies to provide a consistent user experience with a smaller code base to install on those platforms and to maintain. Individual platforms may tailor the size of the architecture to meet their deployment needs and the architecture may be updated independently of the client logic of various applications.

Claims

1. A method for handling user commands via an extensible environment when manipulating visualized data in a user interface, comprising:

receiving raw pointer input from a client;

determining an object selected by the raw pointer input in the user interface based on a position in the user interface of the raw pointer input and a position of the object;

recognizing a gesture based on the raw pointer input, a pointer status, and locations of the raw pointer input relative to the object;

determining an interactivity associated with the object and the gesture; and

transmitting the interactivity to the client to update the user interface.

2. The method of claim 1, wherein the object selected by the raw pointer input includes multiple sub-objects.

3. The method of claim 1, wherein the position in the user interface of the raw pointer input does not match the position of the object and the object is determined to be selected by the raw pointer input as an object closest to the position in the user interface of the raw pointer input.

4. The method of claim 1, wherein a first object is replaced by a second object as the object in response to recognizing the gesture, wherein the gesture is associated with a second object interactivity and not associated with a first object interactivity.

5. The method of claim 1, wherein determining the interactivity is further based on knowing an operating system of the client.

6. The method of claim 1, wherein updating the user interface re-renders the user interface and maintains the layout of the user interface, wherein the interactivity is rendered on a layer above the user interface.

7. The method of claim 6, wherein in response to determining the object selected by the raw pointer input, the object is not rendered on the layer above the user interface and is not rendered in the user interface.

8. The method of claim 1, wherein determining the interactivity is further based on a user interface mode.

9. The method of claim 1, wherein the object is a canvas of the user interface.

10. The method of claim 1, wherein the interactivity is composited with a second interactivity in an interactivity stack for transmission to the client to update the user interface according to the interactivity and the second interactivity.

11. The method of claim 1, wherein the pointer status is chosen from:

a mouse click;

a mouse double-click;

a continuous mouse depression;

pen gestures;

eye tracking gestures;

hover gestures;

touch pad gestures;

a tap touch input;

a double-tap touch input;

a continuous touch input; and

a multi-touch touch input.

12. A system having an architecture for handling user commands via an extensible environment when manipulating visualized data in a user interface, the architecture comprising:

a gesture recognizer; and

an interactivity library;

wherein the gesture recognizer is operable to receive raw pointer input and an object identity from the user interface, wherein the raw pointer input indicates a pointer status and a location in the user interface;

wherein the gesture recognizer is further operable to select an interactivity from the interactivity library based on the raw pointer input and the object identity; and

wherein the interactivity library is operable to transmit the interactivity to update the user interface.

13. The system of claim 12, wherein the gesture recognizer is further operable to receive raw pointer inputs from the user interface on various operating systems to select the interactivity, and wherein the interactivity is operating system agnostic.

14. The system of claim 12, further comprising:

renderers;

data transforms; and

geometry transforms;

wherein the renderers, data transforms, and geometry transforms are modular components that are tailored in size and composition to a client logic providing the user interface.

15. The system of claim 12, further comprising:

an object application program interface in communication with the interactivities library, wherein the object application program interface provides on-object user interface elements related to the interactivity that is transmitted to the user interface.

16. A method for handling user commands via an extensible environment when manipulating visualized data in a user interface, comprising:

transmitting, from an input dispatcher of a client logic, raw pointer input and hit test results to a gesture recognizer hosted in an architecture of the extensible environment, wherein the hit test results identify an object in a document model displayed in the user interface;

receiving, at an interactivity handler hosted in the client logic from the gesture recognizer, an identified gesture based on the raw pointer input and the hit test results;

receiving, at the interactivity handler from an interactivities library hosted in the architecture, an identified action related to the identified gesture to perform on the object;

passing the identified action to a command stack hosted by the client logic; and

executing the identified action passed to the command stack to update the document model and the user interface according to the identified action.

17. The method of claim 16, wherein in cases that the hit test results identify multiple objects, the identified action is directed to a given object of the multiple objects based on the identified gesture being associated with the given object.

18. The method of claim 16, wherein multiple interactivities related to a given object comprise the identified action.

19. The method of claim 18, further comprising:

receiving, via the client logic, an undo command; and

reverting the document model and the user interface to a state prior to executing the identified action.

20. The method of claim 16, wherein the client logic executing the identified action overrides an interactivity performed by the architecture on the identified object.