System and process for controlling a shared display given inputs from multiple users using multiple input modalities

• US 7,458,029 B2
• Filed: 01/15/2004
• Issued: 11/25/2008
• Est. Priority Date: 01/15/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew David, Toyama, Kentaro, Drucker, Steven M., Basu, Sumit, Kristjansson, Trausti Thor, Paek, Tim, Agrawala, Maneesh, Logan, Ronald Keith
• Primary Examiner(s): NGUYEN, CAO H
• Application Number:
  US10/758,574
• Publication Number:
  US 20050198578A1
• Time in Patent Office: 1,776 Days
• Field of Search: 715744-745, 715762-765, 715853-855, 715/760
• US Class Current: 715/744
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A system and process for controlling common information displays, referred to as shared displays, is presented. The system and process allows multiple modes of input using a set of modules that accept and display data from a variety of sources. Input modules are able to understand data from a single mode of communication and to be able to generate messages as output accordingly. An optional translation module takes discrete message units and converts them into commands or requests that can be processed by a logic module. The logic module includes any application that is running on the shared display. A layout module lays out the information output by the logic module and a display module takes the layout data and converts the information to a form that can be readily displayed on a display device.

Claims

1. A system for controlling a shared display, comprising:

a general purpose computing device comprising a display device; and

a computer program comprising program modules executable by the computing device, comprising,

a plurality of input modules each providing a different communication modality, and which collectively input information from multiple users, and wherein each input module comprises a translation sub-module which takes the user-inputted information input into that input module and converts it into commands or requests that are recognizable to a logic module,

the logic module comprising an application which based on the user-inputted information generates display instructions and data, wherein said user-inputted information comprises at least one of video data or audio data or document data,

a layout module which based on the display instructions and data from the logic module generates layout instructions and packages data for display, and

a display module which receives the layout instructions and data from the layout module and employs the same to display content in the shared display on the display device.

2. The system of claim 1, wherein the computer program further comprises a translation module which takes the user-inputted information provided by the input modules and converts it into commands or requests that are recognizable to the logic module, whenever said user-inputted information is in a form not recognizable to the logic module.

3. The system of claim 2, wherein each input module comprises a timestamp sub-module which appends onto each message or a part thereof received from a user that is provided to the translation module, an indicator identifying the time the message was received, and wherein the translation module comprises a sub-module for queuing each message or portion thereof received from the input modules in an order based on its timestamp and providing the messages or parts thereof to the logic module in that order.

4. The system of claim 2, wherein each input module is in communication with each of the other input modules, and wherein each input module comprises a timestamp sub-module which appends onto each message or a part thereof received from a user that is provided to the translation module, an indicator identifying the time the message was received, and wherein each input module comprises a sub-module for coordinating with the other input modules to provide each message or portion thereof to the translation module only after any message or part thereof received by another input module with an earlier timestamp has been provided to the translation module.

5. The system of claim 2, wherein the layout and display modules are specialized so as to support a specific type of display device, and wherein the specialized layout and display modules are swappable with other specialized layout and display modules supporting other display device types such that the layout and display modules which support the type of display employed in the system are employed in lieu of other modules supporting other display device types.

6. The system of claim 2, wherein the translation, layout and display modules are generalized so as to support multiple swappable logic modules each representing a different application.

7. The system of claim 1, wherein the logic module comprises a translation sub-module which takes the user-inputted information provided by the input modules and converts it into commands or requests that are recognizable to the logic module.

8. The system of claim 1, wherein at least one of the input modules comprises an output sub-module which receives data from the logic module and outputs it to a user using the communication modality associated with the input module outputting the data.

9. The system of claim 8, wherein the user specifies in a message input into one of the input modules what data to output from the logic module and which input module having an output sub-module the data is to be output from.

10. The system of claim 8, wherein the data output to the user is at least one of (i) an image file, (ii) a video file, (iii) an audio file, (iv) a document file, or (v) text.

11. The system of claim 1, wherein each input module comprises a timestamp sub-module which appends onto each message or a part thereof received from a user that is provided to the logic module, an indicator identifying the time the message was received, and wherein the logic module comprises a sub-module for scheduling each message or portion thereof received from the input modules to be processed by the logic module based on its timestamp.

12. The system of claim 1, wherein each input module is in communication with each of the other input modules, and wherein each input module comprises a timestamp sub-module which appends onto each message or a part thereof received from a user that is provided to the logic module, an indicator identifying the time the message was received, and wherein each input module comprises a sub-module for coordinating with the other input modules to provide each message or portion thereof to the logic module only after any message or part thereof received by another input module with an earlier timestamp has been provided to the logic module.

13. The system of claim 1, wherein the communication modalities associated with the input modules comprise at least two of (i) email, (ii) short message service (SMS) text messaging, (iii) instant messaging (IM), (iv) DTMF tones, (v) keyboard output signals, (vi) pointing device output signals, (vii) still camera output signals, (viii) video camera output signals and (ix) voice.

14. The system of claim 1, wherein the layout and display modules are generalized so as to support multiple different types of display devices.

15. The system of claim 1, wherein the application associated with the logic module comprising one of (i) a computer game, (ii) an electronic bulletin board, (iii) a voting/polling tool, (iv) a web browsing tool, (v) a computer graphics program or (vi) a lottery tool.

16. The system of claim 1, wherein the display instructions provided by the logic module comprise data priorities specifying which data is to be displayed whenever there is not enough space on the shared display to display all the data provided can be displayed.

17. The system of claim 1, wherein the display instructions provided by the logic module comprise time limits specifying the length of time data displayed on the shared display is to remain displayed.

18. The system of claim 1, wherein the display instructions provided by the logic module comprise hard and soft constraints pertaining to the way the data should be laid out on the shared display.

19. The system of claim 1, wherein the input modules comprises sub-modules for:

parsing messages received from a user into discrete message units according to a prescribed parsing pattern; and

forwarding only those message units to the logic module which are pre-designated as containing information useful to the application associated with the logic module.

20. The system of claim 19, wherein said prescribed parsing plan is designed to separate out an indicator of the identity of the user contained in the message so as to form a message unit which is designated as the user'"'"'s identity.

21. The system of claim 19, wherein said prescribed parsing plan is designed to separate out text or characters representative thereof from the message and form one or more message units which are designated as user text data.

22. The system of claim 19, wherein said prescribed parsing plan is designed to separate out an image from the message and form one or more message units which are designated as user image data.

23. The system of claim 19, wherein said prescribed parsing plan is designed to separate out video from the message and form one or more message units which are designated as user video data.

24. The system of claim 19, wherein said prescribed parsing plan is designed to separate out audio from the message and form one or more message units which are designated as user audio data.

25. The system of claim 19, wherein said prescribed parsing plan is designed to separate out a document file from the message and form one or more message units which are designated as user document data.

26. A computer-implemented process for controlling a shared display on a display device, comprising using a computer to perform the following process actions:

establishing multiple input modalities to input information from multiple users, wherein at least one of the input modalities is characterized by a latency greater than about 1.0 second, and wherein said user-inputted information comprises at least one of video data or audio data or document data; and

inputting the user information from the multiple input modalities to a single computer program which employs the user information to control the content displayed in the shared display on the display device, wherein each input modality translates the user-inputted information input into that input modality and converts it into commands or requests that are recognizable to said single computer program.

27. A system for controlling a shared display, comprising:

a general purpose computing device;

at least one display device showing the shared display; and

a computer program comprising program modules executable by the computing device, comprising,

a plurality of input modules each providing a different input modality at least one of which is characterized by a latency exceeding about 1 second, and which collectively input information from multiple users, wherein said user-inputted information comprises at least one of video data or audio data or document data,

an application module which receives the user information from the input modules and which based on the information generates display layout instructions and packages data for display, and wherein the application module comprises a sub-module for archiving each unique screen shown on the shared display, and

a display module which receives the layout instructions and data from the application module and employs the information and data to display content in the shared display on the display device.

28. The system of claim 27, wherein the application module comprises a sub-module for archiving user-inputted information.

29. The system of claim 27, wherein the application module comprises a sub-module for archiving the identity of each user inputting information to the system, as well as when the information was input and what information was input.

30. The system of claim 27, wherein at least one of the input modules comprises an output sub-module which receives data from the logic module and outputs it to a user using the communication modality associated with the input module outputting the data in response to a user inputted request for the data.

31. The system of claim 30, wherein the application module comprises a sub-module for archiving the identity of each user requesting data, as well as when the information was requested and what data was provided to the user.

32. The system of claim 27, wherein the application module comprises sub-modules for:

determining if the information input by the user comprises a command that has been pre-designated to require special permission to execute;

identifying the user inputting the information and determining if the user is on a pre-established list of user with special permission to submit said command;

executing the command only if the user is on the pre-established list.

33. A computer-readable medium having computer-executable instructions for controlling a shared display on a display device, said computer-executable instructions comprising:

establishing multiple input modalities to input information from multiple users, wherein at least one of the input modalities is characterized by a latency greater than about 1.0 second, and wherein said user-inputted information comprises at least one of video data or audio data or document data;

inputting the user information from the multiple input modalities to a single computer program which employs the user information to control the content displayed in the shared display on the display device; and

outputting data from the computer program to a prescribed group of users using one of more of the communication modalities.