Method and system for robust tagging of named entities in the presence of source or translation errors

• US 10,073,673 B2
• Filed: 07/14/2014
• Issued: 09/11/2018
• Est. Priority Date: 07/14/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Brancovici, George, Nicolae, Gabriel, Luong, Hiep, Vijaya-Raghavan, Sindhu, Lee, Yonghoon
• Primary Examiner(s): Godbold, Douglas
• Assistant Examiner(s): Villena, Mark
• Application Number:
  US14/330,888
• Publication Number:
  US 20160012020A1
• Time in Patent Office: 1,520 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning

Abstract

A system includes a storage device configured to store a plurality of named entities collected from a plurality of sources, tokenized, and associated with a label. The system includes a tagging apparatus configured to receive textual communications, identify each of the textual communications, tokenize the textual communications, match the textual tokens with the named entities tokens stored in the storage device in order to assign the textual tokens to the labels associated with the named entity tokens, tag the textual communications based on the matching between the textual tokens and the named entity tokens in order to identify the intended meaning of each of the textual communications, and identify the intended meaning of the textual communications based on applying the tags to the textual communications. A method capable of disambiguating named entities using a common sense reasoning approach is also disclosed.

Claims

1. An electronic device comprising:

a storage device configured to store a plurality of named entities collected from a plurality of sources, wherein each of the named entities are tokenized into a common format of named entity tokens, wherein each of the named entities are associated with a label, and wherein each of the named entity tokens are one of a word or a syllable of a word; and

one or more processors configured to convert one or more textual communications from a natural language source into a computer readable format for reading and processing by the electronic device, the one or more processors comprising a tagging apparatus configured to;

receive the one or more textual communications,

identify each of the one or more textual communications,

tokenize the one or more textual communications into a common format of textual tokens corresponding to a prefix tree,

match, as a function of a selection of a cluster on a deepest level of a multi-level harmonized clustering structure of tokens, the textual tokens with one or more of the named entity tokens stored in the storage device, in order to assign the textual tokens to the labels associated with each of the named entities,

tag the one or more textual communications based on the matching between the textual tokens and the named entity tokens, in order to identify an intended meaning of each of the one or more textual communications, wherein the intended meaning of the received one or more textual communications are identified based on an extracted conceptual concept from the one or more textual communications,

identify the intended meaning of the one or more textual communications based on applying the tags to the one or more textual communications; and

replace an error of the one or more converted textual communications with the identified intended meaning of the one or more textual communications, when the error is detected.

2. The electronic device as set forth in claim 1, wherein the plurality of named entities comprise at least one of object names, names associated with objects or people, proper names associated with objects or people, titles, phrases, and a combination thereof.

3. The electronic device as set forth in claim 1, wherein the tagging apparatus is configured to apply the tags to the one or more textual communications in order to correct textual mistakes.

4. The electronic device as set forth in claim 1, wherein the tagging apparatus is configured to utilize a beam traversal with a partial solution set of a configurable size on the prefix tree using a custom similarity measure as comparisons occur at sequentially traversed nodes in the prefix tree in order to match the one or more textual communications with the plurality of named entities stored in the storage device.

5. The electronic device as set forth in claim 4, wherein the partial solution set comprises a limited size.

6. The electronic device as set forth in claim 1, wherein the tagging apparatus is configured to partially disambiguate at least one of the plurality of named entities matched with the one or more textual tokens to ensure that a subset of the plurality of named entities of a high-confidence are selected from a full set of plurality of named entities.

7. The electronic device as set forth in claim 1, wherein the tagging apparatus is configured to partially disambiguate at least one of the one or more named entities matched with the one or more textual tokens by filtering out at least one of the one or more named entities having a lowest confidence level of match detection.

8. The electronic device as set forth in claim 1, wherein the tagging apparatus is configured to partially disambiguate at least one of the one or more named entities matched with the one or more textual tokens by ranking confidence levels of at least two of the one or more named entities matched with the one or more textual tokens.

9. A method comprising:

converting, by one or more processors, one or more textual communications from a natural language source into a computer readable format for reading and processing by an electronic device;

receiving, by a tagging apparatus, the converted one or more textual communications;

identifying each of the one or more textual communication;

tokenizing the one or more textual communications into a common format of textual tokens corresponding to a prefix tree;

matching, as a function of a selection of a cluster on a deepest level of a multi-level harmonized clustering structure of tokens, the textual tokens with one or more named entity tokens stored in a storage device, in order to assign the textual tokens to labels associated with the named entity tokens, and wherein each of the one or more named entities is associated with a label;

tagging the one or more textual communications based on the matching between the textual tokens and the named entity tokens, in order to identify an intended meaning of each of the one or more textual communications, wherein the intended meaning of the received one or more textual communications are identified based on an extracted conceptual concept from the one or more textual communications;

identifying the intended meaning of the one or more textual communications based on applying the tags to the one or more textual communications; and

replacing an error of the one or more converted textual communications with the identified intended meaning of the one or more textual communications, when the error is detected.

10. The method as set forth in claim 9, wherein the one or more named entities comprise at least one of object names, names associated with objects or people, proper names associated with objects or people, titles, phrases, and a combination thereof.

11. The method as set forth in claim 9, further comprising applying the tags to the one or more textual communications in order to correct textual mistakes.

12. The method as set forth in claim 9, further comprising utilizing a beam traversal with a partial solution set of a configurable size on the prefix tree using a custom similarity measure as comparisons occur at sequentially traversed nodes in the prefix tree in order to match the one or more textual communications with the one or more named entities stored in the storage device.

13. The method as set forth in claim 12, wherein the partial solution set comprises a limited size.

14. The method as set forth in claim 9, further comprising partially disambiguating at least one of the one or more named entities matched with the one or more textual tokens to ensure that a subset of the one or more named entities of a high-confidence is selected from a full set of one or more named entities.

15. The method as set forth in claim 9, further comprising partially disambiguating at least one of the one or more named entities matched with the one or more textual tokens by filtering out at least one of the one or more named entities having a lowest confidence level of match detection.

16. The method as set forth in claim 9, further comprising partially disambiguating at least one of the one or more named entities matched with the one or more textual tokens by ranking confidence levels of at least two of the one or more named entities matched with the one or more textual tokens.

17. For use in an electronic device, a method comprising:

converting, by one or more processors, one or more textual communications from a natural language source into a computer readable format configured for reading and processing by the electronic device;

receiving the converted one or more textual communications by the electronic device, wherein the one or more textual communications comprises one or more named entities;

grouping, as a function of a selection of a cluster on a deepest level of a multi-level harmonized structure, concepts associated with the one or more name entities in order to extract relevant information from a knowledge base;

inferring relevant knowledge from the extracted relevant information from the knowledge base;

calculating and ranking the inferred relevant knowledge for the one or more named entities; and

identifying a relevant label based on the highest rank of the inferred relevant knowledge for each of the one or more named entities.

18. The method as set forth in claim 17, wherein the one or more textual communications comprises one or more ambiguous entities.

19. The method as set forth in claim 17, wherein the one or more textual communications are preprocessed to remove stop-words and punctuation.

20. The method as set forth in claim 17, wherein calculating and ranking the inferred relevant knowledge includes calculating a score based on the following function: