MULTIMODAL LAYOUT AND RENDERING

• US 20130241951A1
• Filed: 07/05/2012
• Published: 09/19/2013
• Est. Priority Date: 03/16/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dickinson, Richard Lee, Deb, Indrojit, Pallassana, Ramanathan Ananthanarayan, Heydemann, Christian, Paushkina, Anastasia, Cyrek, Ted, McCormack, Michael J., Mondri, Ron, Swaminathan, Bharath, O'Brien, Patrick
• Granted Patent: US 9,310,888 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A layout transformation system accesses application metadata that defines a plurality of different structures for displaying components of the application, based upon the particular modality of the display device. The layout transformation system then transforms a user interface display based on the application metadata and based upon the particular modality, and provides the user interface display to the display device for rendering and user interaction.

Claims

1. A computer-implemented method of generating a representation of data, comprising:

identifying a set of display components;

receiving data bindings that bind the data to the display components;

receiving environmental characteristics indicative of a display modality used to display the data;

receiving component metadata for each of the identified display components; and

generating a user interface display of the identified display components to display the data bound to the identified display components, using the display modality, based on the environmental characteristics and the component metadata.

2. The computer-implemented method of claim 1 and further comprising:

accessing a first transformation template; and

generating the user interface display based on the first transformation template.

3. The computer-implemented method of claim 2 wherein accessing a first transformation template comprises:

accessing one of a plurality of different transformation templates, each transformation template including rendering instructions for rendering display components using a given display modality.

4. The computer-implemented method of claim 3 wherein the plurality of different transformation templates comprise a desktop transformation template that has instructions for rendering a display component using a desktop display modality, a tablet transformation template that has instructions for rendering a display component using a tablet display modality, and a telephone transformation template that has instructions for displaying a display component using a telephone display modality.

5. The computer-implemented method of claim 1 wherein identifying a set of display components comprises:

receiving a user input to display the data; and

receiving the set of display components from a business application.

6. The computer-implemented method of claim 5 and further comprising:

interrogating each given display component in the set of display components to obtain priority metadata indicative of a priority of the given display component, and modality metadata indicative of rendering instructions for rendering the given display component based on the environmental characteristics received.

7. The computer-implemented method of claim 6 wherein generating the user interface display comprises:

generating a layout for the components based on the priority metadata and generating a display of each of the display components based on the modality metadata for each of the display components.

8. The computer-implemented method of claim 7 and further comprising:

receiving a user interaction, during runtime, that changes the environmental characteristics; and

generating the user interface display based on the changed environmental characteristics.

9. The computer-implemented method of claim 7 and further comprising:

identifying a number of canvas sections on a display device based on the environmental characteristics; and

generating the layout for the components based on the number of canvas sections.

10. The computer-implemented method of claim 6 wherein the modality metadata in each display component comprises a plurality of different sets of modality metadata, one set of modality metadata providing rendering instructions for rendering the given display component on a given display modality.

11. The computer-implemented method of claim 10 wherein the plurality of different sets of modality metadata includes desktop metadata that has instructions for rendering the corresponding display component using a desktop display modality, tablet metadata that has instructions for rendering the corresponding display component using a tablet display modality, and telephone metadata that has instructions for displaying the corresponding display component using a telephone display modality.

12. The computer-implemented method of claim 1 wherein receiving the environmental characteristics comprises:

interrogating a display device on which the user interface is displayed for the environmental characteristics.

13. The computer-implemented method of claim 1 wherein the environmental characteristics comprise a type of display device, a size of display screen, a display orientation, and input modalities corresponding to the display device.

14. A business system, comprising:

a layout transformation component that receives environmental characteristics indicative of a display modality used for displaying user interface displays and application metadata indicative of rendering instructions for rendering display components on the display modality, the layout transformation component transforming the user interface display for rendering on one of a plurality of different display modalities based on the environmental characteristics and the application metadata;

a display device displaying the user interface display according to the display modality; and

a computer processor being a functional component of the system and activated by the layout transformation component to facilitate transforming the user interface display.

15. The business system of claim 14 and further comprising:

a business application that identifies particular display components to be rendered and a display priority for the display components.

16. The business system of claim 14 and further comprising:

a transformation template data store storing transformation templates that include rendering instructions for rendering display components according to a plurality of different display modalities.

17. The business system of claim 16 wherein the layout transformation component accesses the transformation data store to obtain a transformation template based on the environmental characteristics.

18. The business system of claim 14 wherein the different display modalities include a touch modality wherein the user interface display is displayed on a display device with a touch sensitive screen.

19. The business system of claim 14 wherein the layout transformation component receives user input that changes the environmental characteristics and transforms the user interface display for rendering on one of the plurality of different display modalities based on the changed environmental characteristics and the application metadata

20. A computer readable storage medium storing computer readable instructions which, when executed by a computer, cause the computer to perform a method, comprising:

identifying a set of display components;

receiving data bindings that bind the data to the display components;

receiving environmental characteristics indicative of a display modality used to display the data;

accessing one of a plurality of different transformation templates, each transformation template including rendering instructions for rendering display components using a given display modality;

receiving component metadata for each of the identified display components; and

generating a user interface display of the identified display components to display the data bound to the identified display components, using the display modality, based on the environmental characteristics the one transformation template, and the component metadata.