Multimodal note taking, annotation, and gaming

• US 20070022372A1
• Filed: 06/29/2005
• Published: 01/25/2007
• Est. Priority Date: 06/29/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Williams, David, Liu, Zicheng, Kurlander, David
• Granted Patent: US 7,694,214 B2
• Time in Patent Office: Days
• US Class Current: 715/201
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

A multimodal, multilanguage mobile device which can be employed to enhance note taking and/or annotation of a document, and gaming. Input data types such as optical character recognition (OCR), speech, handwriting, and visual information (e.g., image and/or video), etc., can be fused to generate rich documents with a multidimensional level of data to provide an increased level of context over conventional documents. Such architecture can be utilized by students for homework management, as well as entertainment (e.g., gaming).

Claims

1. A system that facilitates data processing, comprising:

an input component that receives a plurality of different types of input data; and

an output component that processes the plurality of different types of input data into a document to generate a rich document having a multidimensional level of data.

2. The system of claim 1, wherein one of the plurality of different types of input data processed by the input component is optical character recognition (OCR) data.

3. The system of claim 1, wherein the plurality of different types of input data processed by the input component includes at least one of voice data and audio data.

4. The system of claim 1, wherein the plurality of different types of input data processed by the input component includes at least one of handwriting data and location data.

5. The system of claim 1, wherein the plurality of different types of input data processed by the input component includes image data.

6. The system of claim 1, further comprising an artificial intelligence component that employs a probabilistic and/or statistical-based analysis to prognose or infer an action that a user desires to be automatically performed.

7. A computer-readable medium having stored thereon computer-executable instructions for carrying out the system of claim 1.

8. A portable computing device that employs the system of claim 1 for note-taking, annotation, and gaming.

9. The system of claim 1, further comprising an ordering component that facilitates orderly placement of the plurality of different types of input data into the document according to the time that it was received.

10. The system of claim 1, further comprising an ordering component that facilitates placement of the plurality of different types of input data into the document according to user preferences.

11. The system of claim 1, further comprising a selection component that facilitates selection of one of the plurality of different types of input data based on an application.

12. The system of claim 1, further comprising a multilanguage component that processes speech input and facilitates output of textual content in the rich document that represent the speech input.

13. A computer-implemented method of data processing, the method comprising:

receiving a plurality of different types of input data;

selecting one or more of the plurality of different types of input data according to an application;

associating the one or more of the plurality of different types of input data with a document;

fusing the one or more of the plurality of different types of input data into a fused output with a fusion component; and

outputting a rich document having associated therewith the fused output of the one or more of the plurality of different types of input data.

14. The method of claim 13, further comprising an act of searching a database according to the one or more of the plurality of different types of input data associated with the rich document.

15. The method of claim 13, further comprising an act of launching a compatible application based on the one or more of the plurality of different types of input data selected.

16. The method of claim 13, wherein the one or more of the plurality of different types of input data include at least two of textual data, image data, video data, audio data, speech data, and OCR data.

17. The method of claim 13, further comprising an act of activating video content of the rich document by selecting of an associated video link.

18. The method of claim 13, wherein the further comprising an act of selecting the one or more of the plurality of different types of input data based on a type of the document.

19. The method of claim 13, wherein the further comprising an act of embedding at least one of image data, video data, and audio data in textual data of the rich document.

20. A system that facilitates data processing, comprising:

means for receiving a plurality of different types of input data for document processing;

means for associating the plurality of different types of input data with a document;

means for placing the plurality of different types of input data in the document according to a predetermined order;

means for changing one input data based on analysis of other input data;

means for outputting a rich document having associated therewith the changed input data and the other input data.