Multimodal note taking, annotation, and gaming

• US 7,694,214 B2
• Filed: 06/29/2005
• Issued: 04/06/2010
• Est. Priority Date: 06/29/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kurlander, David, Zhang, Zhengyou, Liu, Zicheng, Williams, David W.
• Primary Examiner(s): Hong; Stephen S
• Assistant Examiner(s): Vaughn; Gregory J
• Application Number:
  US11/172,127
• Publication Number:
  US 20070022372A1
• Time in Patent Office: 1,742 Days
• Field of Search: 715/200, 715/201, 715/202, 715/203, 715/204, 715/230, 715/232, 715/233, 715/273
• US Class Current: 715/203
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

A multimodal, multilanguage mobile device which can be employed to enhance note taking and/or annotation of a document, and gaming. Input data types such as optical character recognition (OCR), speech, handwriting, and visual information (e.g., image and/or video), etc., can be fused to generate rich documents with a multidimensional level of data to provide an increased level of context over conventional documents. Such architecture can be utilized by students for homework management, as well as entertainment (e.g., gaming).

Claims

1. A system that facilitates data processing, comprising:

a processor; and

a memory, on which are stored processor-executable instructions, which when executed by the processor result in operation of an input component and an output component;

wherein the input component receives a plurality of different types of input data;

wherein the output component processes the plurality of different types of input data into a document to generate a rich document having a multidimensional level of data; and

wherein a text document is received by the input component and is processed by an optical character recognition (OCR) subsystem of the input component to produce optically recognized characters, wherein the input component utilizes at least one additional document according to the plurality of different types of input data received, having content that is related to the text document, to verify accuracy of the optically recognized characters, and wherein the verification comprises;

initiating OCR of the text document to generate the optically recognized characters;

determining a likelihood that the optically recognized characters are correct;

accessing the at least one additional document, wherein the at least one additional document is an image document;

analyzing content of the additional document; and

replacing optically recognized characters according to data from the additional document when indicated by the analysis and the likelihood.

2. The system of claim 1, wherein the plurality of different types of input data processed by the input component includes at least one of voice data and audio data.

3. The system of claim 1, wherein the plurality of different types of input data processed by the input component includes at least one of handwriting data and location data.

4. The system of claim 1, wherein the plurality of different types of input data processed by the input component includes image data.

5. The system of claim 1, further comprising an artificial intelligence component that employs a probabilistic and/or statistical-based analysis to prognose or infer an action that a user desires to be automatically performed.

6. A portable computing device that employs the system of claim 1 for note-taking, annotation, and gaming.

7. The system of claim 1, further comprising an ordering component that facilitates orderly placement of the plurality of different types of input data into the document according to the time that it was received.

8. The system of claim 1, further comprising an ordering component that facilitates placement of the plurality of different types of input data into the document according to user preferences.

9. The system of claim 1, further comprising a selection component that facilitates selection of one of the plurality of different types of input data based on an application.

10. The system of claim 1, further comprising a multilanguage component that processes speech input and facilitates output of textual content in the rich document that represent the speech input.

11. A computer-implemented method of data processing, the method comprising:

receiving a plurality of different types of input data;

selecting one or more of the plurality of different types of input data according to an application;

associating the one or more of the plurality of different types of input data with a document;

fusing the one or more of the plurality of different types of input data into a fused output with a fusion component, wherein the fusing of the plurality of different types of input data comprises;

utilizing at least one additional document to verify accuracy of optical character recognition (OCR) of a text document to be fused, and wherein the verification comprises;

initiating OCR of the text document to generate the optically recognized characters;

determining a likelihood that the optically recognized characters are correct;

accessing the at least one other document, wherein the at least one other document is an image document;

analyzing content of the other document; and

replacing optically recognized characters according to data from the other document when indicated by the analysis and the likelihood; and

outputting a rich document having associated therewith the fused output of the one or more of the plurality of different types of input data.

12. The method of claim 11, further comprising an act of searching a database according to the one or more of the plurality of different types of input data associated with the rich document.

13. The method of claim 11, further comprising an act of launching a compatible application based on the one or more of the plurality of different types of input data selected.

14. The method of claim 11, wherein the one or more of the plurality of different types of input data include at least two of textual data, image data, video data, audio data, speech data, and OCR data.

15. The method of claim 11, further comprising an act of activating video content of the rich document by selecting of an associated video link.

16. The method of claim 11, wherein the further comprising an act of selecting the one or more of the plurality of different types of input data based on a type of the document.

17. The method of claim 11, wherein the further comprising an act of embedding at least one of image data, video data, and audio data in textual data of the rich document.

18. A system that facilitates data processing, comprising:

a processor;

a memory, on which are stored processor-executable instructions, which when executed by the processor result facilitates data processing;

means for receiving a plurality of different types of input data for document processing;

means for verifying accuracy of a text document, from among the plurality of different types of input data received by the means for receiving, wherein the text document is processed by an optical character recognition (OCR) algorithm to produce optically recognized characters, wherein at least one additional document having content that is related to the text document is used to verify accuracy of the OCR algorithm, and wherein the verification comprises;

initiating OCR of the text document to generate the optically recognized characters;

determining a likelihood that the optically recognized characters are correct;

accessing the at least one other document, wherein the at least one other document is an image document;

analyzing content of the other document; and

replacing optically recognized characters according to data from the other document when indicated by the analysis and the likelihood;

means for associating the plurality of different types of input data with a document;

means for placing the plurality of different types of input data in the document according to a predetermined order;

means for changing one input data based on analysis of other input data; and

means for outputting a rich document having associated therewith the changed input data and the other input data.