Interactive multimodal display platform

• US 10,289,390 B2
• Filed: 01/05/2015
• Issued: 05/14/2019
• Est. Priority Date: 01/03/2014
• Status: Active Grant

• Current Assignee: Shadowbox Inc.
• Original Assignee: Shadowbox Inc.
• Inventors: Samatov, Jack, Samatov, Lotch
• Primary Examiner(s): Nguyen, Duy Khuong T
• Assistant Examiner(s): Hebert, Theodore E
• Application Number:
  US15/329,564
• Publication Number:
  US 20170315790A1
• Time in Patent Office: 1,590 Days
• Field of Search: 717107
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 8/60 Software deployment
• CPC Class Codes: G06F 8/61 Installation
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

An interactive multimodal display platform comprising a rich Internet application (RIA) built on a cross-platform runtime system that allows a user to combine, connect and customize Web-based applications and local, user-created content for display in a multi-view Presentation. Users can open for a quick view any number of files in different formats from their desktops. The multi-view Presentation may include one or more Acts, with each Act containing up to sixteen Pods in the form of a rotatable three-dimensional cube, with each Pod displaying any file type as selected by the user. All Pods may be interactive with all other Pods within a single Act, and any change to a Pod can trigger corresponding changes in other Pods. After the user has created a customized Act, the Act may be saved as a Rich Interactive Content (RIC) desktop file or saved to the cloud and shared with other users.

Claims

1. A method of displaying multimodal content in a multi-view presentation, the method comprising:

accessing a plurality of Web-based and/or locally-based applications and/or data files;

establishing connections between the plurality of Web-based and/or locally-based applications and/or data files;

accessing a plurality of display files associated with the plurality of accessed applications and/or data files;

generating a Pod for each accessed display file using a template designed for each Pod type, wherein the Pod type is at least partly based on applications and/or data files accessed in the Pod, and the Pod is capable of executing one or more processes associated with the accessed applications and/or data files;

generating an Act, wherein the Act comprises at least one Pod;

connecting Pods within the Act such that processes in Pods within the Act are simultaneously active and related based at least partly on content of the Pods;

receiving a user request for a search;

performing the search across the Pods in the Act and the plurality of Web-based and/or locally-based applications and/or data files;

generating one or more Pods within the Act for each single result of the search;

generating a Presentation, wherein the Presentation comprises one or more Acts; and

displaying the Presentation in the form of a three-dimensional (3D) cube, wherein one or more faces of the cube each comprise an Act displayed in a flat two-dimensional (2D) view; and

receiving user activity in an application accessed and displayed in a Pod and modifying application processes associated with one or more Pods based at least partly on the received user activity.

2. The method of displaying multimodal content of claim 1, wherein generating an Act comprises receiving a user input, wherein the user input includes one or more of:

dragging and dropping files from a desktop computer into the Act;

dragging and dropping regular text with an address or a Web URL into the Act; and

dragging and dropping a text file into the Act.

3. The method of displaying multimodal content of claim 1, further comprising:

receiving a user input to edit an Act in an admin mode, wherein the user input comprises one or more of;

deleting a Pod from the Act;

moving a Pod within the Act;

moving a Pod from the Act to another Act;

changing a Pod from one Pod type to another Pod type;

changing display content of a Pod; and

editing and displaying the Act based on the user input.

4. The method of displaying multimodal content of claim 1, further comprising the step of communicating between Pods of discrepant types of an Act by exchanging data between the Pods in Extensible Markup Language (XML) or JavaScript Object Notation (JSON) format.

5. The method of displaying multimodal content of claim 1, wherein the Pod type comprises a Single Media Pod, a Photo Viewer Pod, a Video Player Pod, a Web Browser Pod, a PowerPoint Pod, a Flash Animation (SWF) Pod, a Chart &

Graph Pod, an HDF5 Data Viewer Pod, a PDF Viewer Pod, a Live Video Streaming Pod, a GoogleMaps Pod, and an Audio Player Pod.

6. An interactive multimodal display platform for providing customizable multi-view presentations, the interactive multimodal display platform comprising:

a Rich Internet Application (RIA) configured to connect with Web-based applications and locally-based data;

a computer system configured to support the RIA by interfacing between the RIA and the Web-based applications and/or locally-based data;

a compound display module configured by the RIA to display content from the Web-based applications and/or locally-based data in a single, interactive customizable multi-view display, wherein the multi-view display comprises one or more Acts, wherein an Act comprises one or more Pods and the RIA is configured to execute and/or display a program segment in each Pod, the program segment associated with, or at least partly based on, the accessed Web-based applications and/or locally-based data, and the RIA is further configured to;

receive a user request for a search, perform the search across Pods in an Act and the accessed Web-based applications and/or locally-based data,

generate one or more Pods within the Act for each single result of the search;

generate a Presentation, wherein the Presentation comprises one or more Acts;

display the Presentation in the form of a three-dimensional (3D) cube, wherein one or more faces of the cube each comprise an Act displayed in a flat two-dimensional (2D) view; and

a connected view module configured to enable a dynamic connection between the Pods within an Act, wherein a user-triggered event in a first Pod in a first Act triggers the RIA to perform one or more contextually related actions and/or display results in second, third or more Pods within the first Act.

7. The interactive multimodal display platform of claim 6, wherein the RIA is further configured to:

receive a user input to edit an Act in an admin mode, wherein the user input comprises one or more of;

delete a Pod from the Act;

move a Pod within the Act;

move a Pod from the Act to another Act;

change a Pod from one Pod type to another Pod type;

change the content of the Pod; and

editing and displaying the Act based on the user input.

8. The interactive multimodal display platform of claim 6, wherein the one or more contextually related actions comprise:

one or more of searching for contents and/or programs related to the triggering event locally and/or on the Web and executing web-based and/or local programs contextually related to the triggering event.

9. The interactive multimodal display platform of claim 6, wherein the contextually related actions comprise:

auto-generating content in the second, third and/or more Pods of the first Act based on content displayed in the first Pod of the first Act and/or based on receiving user input in the first Pod of the first Act.

10. The interactive multimodal display platform of claim 9, wherein the user input comprises receiving an input from a user dragging and dropping content from the first Pod to the second Pod.

11. A method of displaying results of a single-topic search in a multi-view presentation, the method comprising:

establishing connections between and accessing a plurality of Web-based and/or locally-based data files;

generating one or more Acts, wherein Acts comprise one or more Pods, wherein each Pod is configured to execute a program segment and display content based on an accessed data file;

dynamically connecting Pods within an Act, wherein an event in one Pod triggers contextually related events in other Pods;

in a first Pod of a first Act, receiving a user request for a single-topic search;

simultaneously searching for the single-topic in Pods within the first Act and the plurality of Web-based and/or locally-based data files;

determining one or more categories of the single-topic search;

generating additional search terms contextually related to the single-topic search, based at least partly on the one or more categories of the single-topic search;

determining additional web-based and/or locally-based sources to search for the single-topic and additional search terms;

searching for the single-topic and/or additional search terms in one or more of;

contents of Pods within the first Act, web-based and/or locally-based data files and the determined additional Web-based and/or locally-based sources;

generating one or more Pods within the first Act for each single result of the search; and

displaying the first Act in a multi-view presentation, wherein the multi-view presentation is in the form of a three-dimensional (3D) cube, wherein one or more faces of the cube each comprise an Act displayed in a flat two-dimensional (2D) view.

12. The method of displaying results of a single-topic search of claim 11, wherein displaying the first Act in a multi-view presentation further comprises displaying each of the search results in a Pod within the first Act, wherein each Pod comprises its own window within the multi-view presentation.

13. The method of displaying results of single-topic search of claim 11, further comprising:

modifying the search results by receiving an input from a user, the user input comprising dragging and dropping more content from a desktop computer or a Web URL;

customizing, saving, and sharing the search results;

retrieving relative content from the new content added into the search results; and

displaying the results of the new search in the multi-view presentation.

14. The method of displaying results of single-topic search of claim 11, further comprising:

reading metadata from media files in media Pods;

putting a pushpin on a map within a map Pod representing a location; and

connecting locations with location-based data of any Web source if added into an Act with a map showing a location.