Multimodal layout and rendering

• US 9,310,888 B2
• Filed: 07/05/2012
• Issued: 04/12/2016
• Est. Priority Date: 03/16/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Dickinson, Richard Lee, Deb, Indrojit, Pallassana, Ramanathan Ananthanarayan, Heydemann, Christian, Paushkina, Anastasia, Cyrek, Ted, Swaminathan, Bharath, McCormack, Michael J., Mondri, Ron, O'Brien, Patrick
• Primary Examiner(s): Du, Haixia
• Application Number:
  US13/541,785
• Publication Number:
  US 20130241951A1
• Time in Patent Office: 1,377 Days
• Field of Search: 345/619, 715234-253, 715273-277, 715/760, 715762-763, 715/864, 715/867, 715/772
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A layout transformation system accesses application metadata that defines a plurality of different structures for displaying components of the application, based upon the particular modality of the display device. The layout transformation system then transforms a user interface display based on the application metadata and based upon the particular modality, and provides the user interface display to the display device for rendering and user interaction.

Claims

1. A computer-implemented method of generating a representation of application data, comprising:

identifying a plurality of display components for a user interface display;

receiving an indication of data bindings that bind the application data to the display components;

receiving an indication of an environmental characteristic, indicative of a particular display modality used to display the application data;

for each display component in the plurality of display components,

interrogating the display component for component metadata that includes priority metadata and modality metadata, the priority metadata indicating a priority of the display component relative to other display components, and the modality metadata indicating how to render the display component for the particular display modality;

identifying a set of canvas sections for rendering the user interface display using the particular display modality, wherein each canvas section has an associated display priority; and

generating the user interface display for display on a display device, wherein generating comprises;

for each of the canvas sections, selecting a display component from the plurality of display components based on the display priority of the canvas section and the priority of the selected display component indicated by the priority metadata, wherein the selected display component displays the application data bound to the selected display component based on the modality metadata of the selected display component.

2. The computer-implemented method of claim 1 wherein generating the user interface display comprises:

selecting a first display component having a highest priority for display in a first canvas section having a highest display priority.

3. The computer-implemented method of claim 1, further comprising:

selecting a first transformation template from a plurality of different transformation templates that correspond to a plurality of different display modalities, each transformation template of the plurality of different transformation templates including rendering instructions for rendering display components using a given one of the different display modalities.

4. The computer-implemented method of claim 3 wherein the plurality of different transformation templates are distinct from the priority and modality metadata for the display components, and wherein the plurality of different transformation templates comprise:

a desktop transformation template that has instructions for rendering a display component using a desktop display modality,

a tablet transformation template that has instructions for rendering a display component using a tablet display modality, and

a telephone transformation template that has instructions for displaying a display component using a telephone display modality.

5. The computer-implemented method of claim 1 wherein identifying a plurality of display components comprises:

receiving an indication of a user input to display the application data; and

receiving the set of display components from an application.

6. The computer-implemented method of claim 1, wherein the modality metadata for each display component is indicative of rendering instructions for rendering the display component on a plurality of different modalities.

7. The computer-implemented method of claim 1, wherein generating the user interface display comprises:

generating a layout for the display components based on the priority metadata and generating a display of each of the display components based on the modality metadata for each of the display components.

8. The computer-implemented method of claim 1, and further comprising:

receiving an indication of a user interaction, during runtime, that changes the environmental characteristic; and

generating the user interface display based on the changed environmental characteristic.

9. The computer-implemented method of claim 1 wherein the component metadata for each display component comprises a plurality of different sets of modality metadata for the display component, each set of modality metadata indicating rendering instructions for rendering the display component on a different display modality.

10. The computer-implemented method of claim 9 wherein the plurality of different sets of modality metadata includes desktop metadata that has instructions for rendering the corresponding display component using a desktop display modality, tablet metadata that has instructions for rendering the corresponding display component using a tablet display modality, and telephone metadata that has instructions for displaying the corresponding display component using a telephone display modality.

11. The computer-implemented method of claim 9, and further comprising:

selecting one of the different sets of modality metadata, for each display component, based on the display modality indicated by the environmental characteristic; and

generating the user interface display based on the selected set of modality metadata.

12. The computer-implemented method of claim 1 wherein the environmental characteristic comprise a type of display device, a size of display screen, a display orientation, and input modalities corresponding to the display device, and wherein receiving an indication of the environmental characteristic comprises:

interrogating a display device on which the user interface display is displayed for the environmental characteristic.

13. A computing system, comprising:

a layout transformation component configured to;

receive an environmental characteristic that indicates a display modality used for displaying a user interface display on a display device;

receive application metadata for an application having a plurality of display components, wherein the application metadata is indicative of rendering instructions for rendering each of the display components and comprises component priority metadata that indicates a display priority for the plurality of display components;

define a layout of the user interface display for rendering on the display modality, the layout comprising a canvas section configured to sequentially display the plurality of display components on the user interface display in response to a user input, wherein the layout transformation component defines an order for the sequential display of the plurality of display components based on the display priority for the plurality of display components; and

a user interface component configured to instruct the display device to display the user interface display.

14. The computing system of claim 13 and further comprising:

an application configured to identify particular display components to be rendered and a display priority for the display components.

15. The computing system of claim 13 and further comprising:

a transformation template data store configured to store transformation templates that include rendering instructions for rendering display components according to a plurality of different display modalities.

16. The computing system of claim 15 wherein the layout transformation component is configured to access the transformation template data store to obtain a particular one of the transformation templates, based on the environmental characteristic, and use the particular transformation template to render the plurality of display components.

17. The computing system of claim 13 wherein the display device comprises a touch-sensitive display device, the display modality comprises a touch modality used to display the user interface display on the touch-sensitive display device, and the user input comprises a user swipe gesture on the touch-sensitive display device.

18. The computing system of claim 17, wherein the user interface component is configured to:

instruct the touch-sensitive display device to display a first one of the display components with a highest display priority;

receive an indication of the user swipe gesture; and

instruct the touch-sensitive display device to switch from the display of the first display component to display a second one of the display components with a second highest display priority.

19. A computer-implemented method comprising:

identifying a set of display components;

receiving an indication of data bindings that bind data to the display components;

receiving an indication of an environmental characteristic indicative of a given display modality used to display the data;

selecting, based on the environmental characteristic indicative of the given display modality, a transformation template from a plurality of different transformation templates, each different transformation template including rendering instructions for rendering display components in a particular layout using a different display modality;

identifying a set of canvas sections for rendering the user interface display using the given display modality, and each canvas section has an associated display priority;

receiving component metadata, that is separate from the selected transformation template, for each of the identified display components, the component metadata including priority metadata indicating a priority of the identified display component;

defining a layout for a user interface display of the identified display components that displays the data bound to the identified display components, wherein defining the layout comprises selecting, for each of the canvas sections, a display component from the set of display components based on the display priority of the canvas section and the priority of the selected display component indicated by the priority metadata; and

generating rendering instructions for the user interface display.