Extracting structured data from handwritten and audio notes

• US 10,691,885 B2
• Filed: 03/30/2017
• Issued: 06/23/2020
• Est. Priority Date: 03/30/2016
• Status: Active Grant

• Current Assignee: Bending Spoons SpA
• Original Assignee: Evernote Corp. (Bending Spoons SpA)
• Inventors: O'Neill, Chris, Malcolm, Andrew, Pashintsev, Alexander, Livshitz, Eugene, Gorbatov, Boris, Galaktionova, Natalia, Buryak, Ilya
• Primary Examiner(s): Thai, Hanh B
• Application Number:
  US15/475,001
• Publication Number:
  US 20170286529A1
• Time in Patent Office: 1,181 Days
• Field of Search: 707722
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06F 40/279 Recognition of textual entities

Abstract

This application is directed to recognizing unstructured information based on hints provided by structured information. A computer system obtains unstructured information collected from a handwritten or audio source, and identifies one or more terms from the unstructured information. The one or more terms includes a first term that is ambiguous. The computer system performs a recognition operation on the first term to derive a first plurality of candidate terms for the first term, and obtains first contextual information from an information template associated with the unstructured information. In accordance with the first contextual information, the computer system selects a first answer term from the first plurality of candidate terms, such that the first term is recognized as the first answer term.

Claims

1. A computer-implemented method, comprising:

at a computer system having one or more processors and memory storing one or more programs executed by the one or more processors;

obtaining unstructured information collected from a handwritten or audio source;

identifying one or more terms from the unstructured information, the one or more terms including a first term that is ambiguous;

performing a recognition operation on the first term to derive a first plurality of candidate terms for the first term;

obtaining first contextual information from an information template associated with the unstructured information;

in accordance with the first contextual information, selecting a first answer term from the first plurality of candidate terms, such that the first term is recognized as the first answer term;

obtaining second contextual information from a parallel source that is independent from the handwritten or audio source and the information template; and

verifying that the first term has been properly recognized as the first answer term based on the second contextual information.

2. The method of claim 1, wherein the first plurality of candidate terms are corresponding to different segmentations or recognitions of the first term.

3. The method of claim 1, the first contextual information including a plurality of contextual options, wherein the first answer term at least partially matches one of the plurality of contextual options, and other candidate terms of the first plurality of candidate terms do not match any of the plurality of contextual options.

4. The method of claim 1, wherein:

the information template further comprises a plurality of data units, each data unit having a field name and a form field associated with the field name, the plurality of data units further including a first data unit;

the first contextual information relates to the form field of the first data unit; and

the first answer term recognized from the first term is used to define the form field of the first data unit.

5. The method of claim 4, wherein the form field corresponds to a plurality of predefined menu entries, and the first answer term at least partially matches one of the plurality of predefined menu entries, while other candidate terms of the first plurality of candidate terms do not match any of the plurality of predefined menu entries.

6. The method of claim 5, wherein the plurality of predefined menu entries are predefined and stored in a database.

7. The method of claim 5, wherein the plurality of predefined menu entries includes two or more company names.

8. The method of claim 5, wherein the one of the plurality of predefined menu entries includes a first name, and the first answer term at least partially matches one of the plurality of predefined menu entries, when the first answer term matches the first name or a variation of the first name, the variation of the first name including one of a group consisting of a synonym, an abbreviation, a homograph and a homophone of the first name.

9. The method of claim 1, wherein the one or more terms further includes a second term located in proximity to the first term in the unstructured information, and selecting the first answer term from the first plurality of candidate terms further comprises:

recognizing the second term as a second answer term;

determining that the second answer term is related to the first contextual information and the first answer term of the first plurality of candidate terms; and

in accordance with the first contextual information, recognizing the first term as the first answer term.

10. The method of claim 9, the first contextual information including a first name, wherein the second term is associated with the second answer term when the second answer term matches the first name or a variation of the first name, the variation of the first name including one of a group consisting of a synonym, an abbreviation, a homograph and a homophone of the first name.

11. The method of claim 1, wherein:

the information template further comprises a plurality of data units, each data unit having a field name and a form field associated with the field name, the plurality of data units further including a first data unit;

the first contextual information includes the field name of the first data unit; and

the first answer term recognized from the first term is used to define the form field of the first data unit.

12. The method of claim 11, wherein the one or more terms further includes a second term located in proximity to the first term in the unstructured information, and selecting the first answer term from the plurality of candidate terms further comprises:

recognizing the second term as a second answer term;

after determining that the second answer term matches the field name of the first data unit or a variation of the field name of the first data unit, determining that the form field is associated with the first term; and

recognizing the first term as the first answer term.

13. The method of claim 12, wherein the variation of the field name of the first data unit includes one of a group consisting of a synonym, an abbreviation, a homograph and a homophone of the field name of the first data unit.

14. The method of claim 1, wherein selecting a first answer term from the first plurality of candidate terms further comprises:

in accordance with the first contextual information, identifying the first answer term and a second answer term that is recognized as the first term;

displaying a user interface to present the first and second candidate terms; and

receiving a user selection of the first answer term, such that the first term is recognized as the first answer term based on the user selection.

15. A computer system, comprising:

one or more processors; and

memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for;

obtaining unstructured information collected from a handwritten or audio source;

identifying one or more terms from the unstructured information, the one or more terms including a first term that is ambiguous;

performing a recognition operation on the first term to derive a first plurality of candidate terms for the first term;

obtaining first contextual information from an information template associated with the unstructured information;

in accordance with the first contextual information, selecting a first answer term from the first plurality of candidate terms, such that the first term is recognized as the first answer term;

obtaining second contextual information from a parallel source that is independent from the handwritten or audio source and the information template; and

verifying that the first term has been properly recognized as the first answer term based on the second contextual information.

16. The computer system of claim 15, wherein the instructions for selecting a first answer term from the first plurality of candidate terms further comprises instructions for:

in accordance with the first contextual information, identifying the first answer term and a second answer term that is recognized as the first term;

displaying a user interface to present the first and second candidate terms; and

receiving a user selection of the first answer term, such that the first term is recognized as the first answer term based on the user selection.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system, the one or more programs comprising instructions for:

obtaining unstructured information collected from a handwritten or audio source;

identifying one or more terms from the unstructured information, the one or more terms including a first term that is ambiguous;

performing a recognition operation on the first term to derive a first plurality of candidate terms for the first term;

obtaining first contextual information from an information template associated with the unstructured information;

in accordance with the first contextual information, selecting a first answer term from the first plurality of candidate terms, such that the first term is recognized as the first answer term;

obtaining second contextual information from a parallel source that is independent from the handwritten or audio source and the information template; and

verifying that the first term has been properly recognized as the first answer term based on the second contextual information.

18. The non-transitory computer readable storage medium of claim 17, wherein:

the information template further comprises a plurality of data units, each data unit having a field name and a form field associated with the field name, the plurality of data units further including a first data unit;

the first contextual information includes one of the field name and the form field of the first data unit; and

the first answer term recognized from the first term is used to define the form field of the first data unit.